In my previous posts I wrote about my set-up of MariaDB Galera on Kubernetes. Now I have some first experience with this set-up and can provide some guidance. I used an ill-fated TCP health-check that lead to MariaDB Galera blocking the originating IPv4 address from accessing the cluster due to never completing a MySQL handshake and it seems (logs are gone) that this lead to the sync between different systems breaking too.
When I woke up my entire cluster was down and didn’t recover. Some pods restarted and I run into a Azure Kubernetes bug where a Persistent Storage would be umounted but not detached. This means the storage can not be re-attached to the new pod. The Microsoft upstream project is a bit hostile but the issue is known. If you are seeing an error about the storage still being detached/attached. You can go to the portal, find the agent that has it attached and detach it by hand.
To bring the cluster back online there is a chicken/egg problem. The entrypoint.sh discovers the members of the cluster by using environment variables. If the cluster is entirely down and the first pod is starting, it will just exit as it can’t connect to the others. My first approach was to keep the other nodes down and use kubectl edit rc/galera-node-X and set replicas to 0. But then the service is still exporting the information. In the end I deleted the srv/galera-node-X and waited for the first pod to start. Once it was up I could re-create the services again.
My next steps are to add proper health checks, some monitoring and see if there is a more long term archive for the log data of a (deleted) pod.
As I blogged in my blog post in Fabruary, I was working towards a more
fully-featured SIGTRAN stack in the Osmocom (C-language) universe.
The trigger for this is the support of 3GPP compliant AoIP (with a
BSSAP/SCCP/M3UA/SCTP protocol stacking), but it is of much more general
The code has finally matured in my development branch(es) and is now
ready for mainline inclusion. It's a series of about 77 (!) patches,
some of which already are the squashed results of many more incremental
The result is as follows:
- General SS7 core functions maintaining links, linksets and routes
- xUA functionality for the various User Adaptations (currently SUA and M3UA supported)
- MTP User SAP according to ITU-T Q.701 (using osmo_prim)
- management of application servers (AS)
- management of application server processes (ASP)
- ASP-SM and ASP-TM state machine for ASP, AS-State Machine (using osmo_fsm)
- server (SG) and client (ASP) side implementation
- validated against ETSI TS 102 381 (by means of Michael Tuexen's m3ua-testtool)
- support for dynamic registration via RKM (routing key management)
- osmo-stp binary that can be used as Signal Transfer Point, with
the usual "Cisco-style" command-line interface that all Osmocom
telecom software has.
- SCCP implementation, with strong focus on Connection Oriented SCCP (as
that's what the A interface uses).
- osmo_fsm based state machine for SCCP connection, both incoming and
- SCCP User SAP according to ITU-T Q.711 (osmo_prim based)
- Interfaces with underlying SS7 stack via MTP User SAP (osmo_prim based)
- Support for SCCP Class 0 (unit data) and Class 2 (connection oriented)
- All SCCP + SUA Address formats (Global Title, SSN, PC, IPv4 Address)
- SCCP and SUA share one implementation, where SCCP messages are
transcoded into SUA before processing, and re-encoded into SCCP after
processing, as needed.
I have already done experimental OsmoMSC and OsmoHNB-GW over to libosmo-sigtran.
They're now all just M3UA clients (ASPs) which connect to osmo-stp
to exchange SCCP messages back and for the between them.
What's next on the agenda is to
- finish my incomplete hacks to introduce IPA/SCCPlite as an alternative
to SUA and M3UA (for backwards compatibility)
- port over OsmoBSC to the SCCP User SAP of libosmo-sigtran
- validate with SSCPlite lower layer against existing SCCPlite MSCs
- implement BSSAP / A-interface procedures in OsmoMSC, on top of the
If those steps are complete, we will have a single OsmoMSC that can talk
both IuCS to the HNB-GW (or RNCs) for 3G/3.5G as well as AoIP towards
OsmoBSC. We will then have fully SIGTRAN-enabled the full Osmocom
stack, and are all on track to bury the OsmoNITB that was devoid of such
If any reader is interested in interoperability testing with other
implementations, either on M3UA or on SCCP or even on A or Iu interface
level, please contact me by e-mail.
In my previous post I wrote about getting a MariaDB Galera cluster started on Kubernetes. One of my open issues was how to get my existing VM to connect to it. With Microsoft Azure the first thing is to add Network peering between the Kubernetes cluster and the normal VM network. As previously mentioned the internal IPv4 address of the Galera service is not reachable from outside and the three types of exposing a service are:
While the default Microsoft Azure setup already has two LoadBalancers, the kubectl expose –type=LoadBalancer command does not seem to allow me to chose which load balancer to use. So after trying this command my Galera cluster was reachable through a public IPv4 address on the standard MySQL port. While it is password protected it didn’t seem like a good idea. To change the config you can use something like kubectl edit srv/galera-cluster and change the type to another one. Then I tried the NodePort type and got the MySQL port exposed on all masters and thanks to the network peering was able to connect to them directly. Then I manually modified the already configured/created Microsoft Azure LoadBalancer for the three masters to export port 3306 and map it to the internal port. I am also doing a basic health check which checks if port 3306 can be connected to.
Now I can start using the Galera cluster from my container based deployment before migrating it fully to Kubernetes. My next step is probably to improve the health checks to only get primaries listed in the LoadBalancer and then add monitoring to it as well.
As part of my journey to “cloud” computing I built a service that is using MySQL and as preparation for the initial deployment I set myself the following constraints:
- Deploy in containers
- Be able to tolerate some failure of ” VM”s
- Be able to grow/replace storage without downtime
There are pre-made mariadb:10.1 containers but to not rely on a public registry I have used the Microsoft Azure Container Service to upload my container. The integration into the standard docker tools to create and upload containers just worked. It allows me to give a place for modified containers as well.
With Azure it doesn’t seem possible to online resize (grow) a volume and if I ever want to switch from ext4 to xfs (or zfs?) I should run some form of fault tolerant MySQL to take a node and upgrade it. These days MariaDB 10.1 includes Galera support and besides some systematic issues (which I don’t seem to run in as I have little to no transactions) it seems quite easy to set-up.
Fault tolerance comes in a couple flavors. Galera is a multi-master database where the cluster will continue to allow writes as long as there is a majority of active nodes. If I start with three nodes, I can take one off the cluster to maintain.
Kubernetes will reschedule a pod/container to a different machine (“agent”) in case one becomes unhealthy and it will expose the Galera cluster through a LoadBalancer and a single IPv4 address for it. This means only active members of the cluster will be contacted.
The last part is provided by Microsoft Azures availability set. Distributing the Agents into different zones should prevent all of them to go down at the same time during maintenance.
So in theory this looks quite nice, only practice will tell how this will play out.
After having picked Microsoft Azure, Kubernetes and Galera, it is time to set it up. I have started with an example found here. I had to remove some labels to make it work with the current format, moved the container to mariadb:10.1 and modified the default config.
I had to look a bit on how to get persistent storage. I am directly mounting the disk for the pod an alternative is a persistent volume claim. This might be a better approach.
The biggest issue is starting the first service. It requires to pass special parameters to initialize the cluster and involved a round of kubectl edit/kubectl delete to get it up. Having the second and third member join was more easy.
Besides having to gain more experience with it, I do face a couple of problems with this setup and need to explore solutions (or wait for comments?).
I deployed my application before having a Kubernetes cluster and now need to migrate. The default networking of Kubernetes works by adding a lot of masquerading entries on agents and masters. In the cluster these addresses are routable by masquerading but from external they are not reachable. I need to find a way to access it, probably by sacrificing some redundancy first. The other option is to use kubectl expose but I don’t want my cluster to have a public IPv4 address. I need to see how to have an internal load balancer with a private/internal IPv4 address.
Galera cluster management is a bit troubling. The first time I start with a new disk it will not properly connect to the master but would register itself to the LoadBalancer/Service. I manually need to do a kubectl delete of the pod and wait for it to reschedule. That is probably easy to fix. The second part of the problem is that I should use health checks and only register the pod once it has connected and synced to the primaries.
Rolling upgrades seem to have a systematic issue too. The default way for the built-in replication controller looks like a new pod (N+1) will be launched and brought up and then the current galera node will be stopped (back to N). This falls apart with the way I mount the storage/disk. E.g. the new pod can not mount the disk as it is already mounted and the old pod will not be deleted.
Least problematic is auto-scaling. In the example set-up each node is a service by itself, using one persistent disk. It makes scaling the cluster a bit difficult. I can add new nodes and they will discover the master(s) but to have the masters remember the new nodes, I would need to have the pods recycle.
April 21st is approaching fast, so here some updates. I'm particularly
happy that we now have travel grants available. So if the travel
expenses were preventing you from attending so far: This excuse is no
Get your ticket now, before it is too late. There's a limited number of
OsmoCon 2017 Schedule
The list of talks
for OsmoCon 2017 has been
available for quite some weeks, but today we finally published the first
As you can see, the day is fully packed with talks about Osmocom
cellular infrastructure projects. We had to cut some talk slots short
(30min instead of 45min), but I'm confident that it is good to cover a
wider range of topics, while at the same time avoiding fragmenting the
audience with multiple tracks.
OsmoCon 2017 Travel Grants
We are happy to announce that we have received donations to permit for
providing travel grants!
This means that any attendee who is otherwise not able to cover their
travel to OsmoCon 2017 (e.g. because their interest in Osmocom is not
related to their work, or because their employer doesn't pay the travel
expenses) can now apply for such a travel grant.
For more details see OsmoCon 2017 Travel Grants
and/or contact email@example.com.
Back in October 2016 I designed a small open hardware breakout board
for WWAN modems in mPCIe form-factor. I was thinking some
other people might be interested in this, and indeed, the first
manufacturing batch is already sold out by now.
Instead of ordering more of the old (v2) design, I decided to do some
improvements in the next version:
- add mounting holes so the PCB can be mounted via M3 screws
- add U.FL and SMA sockets, so the modems are connected via a short U.FL
to U.FL cable, and external antennas or other RF components can be
attached via SMA. This provides strain relief for the external
antenna or cabling and avoids tearing off any of the current loose
U.FL to SMA pigtails
- flip the SIM slot to the top side of the PCB, so it can be accessed
even after mounting the board to some base plate or enclosure via the
- more meaningful labeling of the silk screen, including the purpose of
the jumpers and the input voltage.
A software rendering of the resulting v3 PCB design files that I just
sent for production looks like this:
Like before, the design of the board (including schematics and PCB
layout design files) is available as open hardware under CC-BY-SA
license terms. For more information see
It will take some expected three weeks until I'll see the first
I'm also planning to do a M.2 / NGFF version of it, but haven't found
the time to get around doing it so far.
As I just wrote in my post about TelcoSecDay, I sometimes
worry about the choices I made with Osmocom, particularly when I see
all the great stuff people doing in fields that I previously was working
in, such as applied IT security as well as Linux Kernel development.
When people like Dieter, Holger and I started to play with what later
became OpenBSC, it was just for fun. A challenge to master. A closed
world to break open and which to attack with the tools, the mindset and
the values that we brought with us.
Later, Holger and I started to do freelance development for commercial
users of Osmocom (initially basically only OpenBSC, but then OsmoSGSN,
OsmoBSC, OsmoBTS, OsmoPCU and all the other bits on the infrastructure
side). This lead to the creation of sysmocom in 2011, and ever since we
are trying to use revenue from hardware sales as well as development
contracts to subsidize and grow the Osmocom projects. We're investing
most of our earnings directly into more staff that in turn works on
Osmocom related projects.
It's important to draw the distinction betewen the Osmocom cellular
which are mostly driven by commercial users and sysmocom these days,
and all the many other pure juts-for-fun community projects under
the Osmocom umbrella, like OsmocomTETRA, OsmocomGMR, rtl-sdr, etc.
I'm focussing only on the cellular infrastructure projects, as they
are in the center of my life during the past 6+ years.
In order to do this, I basically gave up my previous career[s] in IT
security and Linux kernel development (as well as put things like
gpl-violations.org on hold). This is a big price to pay for crating
more FOSS in the mobile communications world, and sometimes I'm a bit
melancholic about the "old days" before.
Financial wealth is clearly not my primary motivation, but let me be
honest: I could have easily earned a shitload of money continuing to do
freelance Linux kernel development, IT security or related consulting.
There's a lot of demand for related skills, particularly with some
experience and reputation attached. But I decided against it, and
worked several years without a salary (or almost none) on Osmocom
related stuff [as did Holger].
But then, even with all the sacrifices made, and the amount of revenue
we can direct from sysmocom into Osmocom development: The complexity of
cellular infrastructure vs. the amount of funding and resources is always
only a fraction of what one would normally want to have to do a proper
implementation. So it's constant resource shortage, combined with lots
of unpaid work on those areas that are on the immediate short-term
feature list of customers, and that nobody else in the community feels
like he wants to work on. And that can be a bit frustrating at times.
Is it worth it?
So after 7 years of OpenBSC, OsmocomBB and all the related projects, I'm
sometimes asking myself whether it has been worth the effort, and
whether it was the right choice.
It was right from the point that cellular technology is still an area
that's obscure and unknown to many, and that has very little FOSS
(though Improving!). At the same time, cellular networks are becoming
more and more essential to many users and applications. So on an
abstract level, I think that every step in the direction of FOSS for
cellular is as urgently needed as before, and we have had quite some
success in implementing many different protocols and network elements.
Unfortunately, in most cases incompletely, as the amount of funding
and/or resources were always extremely limited.
On the other hand, when it comes to metrics such as personal
satisfaction or professional pride, I'm not very happy or satisfied.
The community remains small, the commercial interest remains limited,
and as opposed to the Linux world, most players have a complete lack of
understanding that FOSS is not a one-way road, but that it is important
for all stakeholders to contribute to the development in terms of
I think a collaborative development project (which to me is what FOSS is
about) is only then truly successful, if its success is not related to
a single individual, a single small group of individuals or a single
entity (company). And no matter how much I would like the above to be
the case, it is not true for the Osmocom cellular infrastructure
projects. Take away Holger and me, or take away sysmocom, and I think
it would be pretty much dead. And I don't think I'm exaggerating here.
This makes me sad, and after all these years, and after knowing quite a
number of commercial players using our software, I would have hoped that
the project rests on many more shoulders by now.
This is not to belittle the efforts of all the people contributing to
it, whether the team of developers at sysmocom, whether those in the
community that still work on it 'just for fun', or whether those
commercial users that contract sysmocom for some of the work we do.
Also, there are known and unknown donors/funders, like the NLnet
foundation for some parts of the work. Thanks to all of you, and
clearly we wouldn't be where we are now without all of that!
But I feel it's not sufficient for the overall scope, and it's not [yet]
sustainable at this point. We need more support from all sides,
particularly those not currently contributing. From vendors of BTSs and
related equipment that use Osmocom components. From operators that use
it. From individuals. From academia.
Yes, we're making progress. I'm happy about new developments like the
Iu and Iuh support, the OsmoHLR/VLR split and 2G/3G authentication that Neels just blogged about. And
there's progress on the SIMtrace2 firmware with card emulation and MITM,
just as well as there's progress on libosmo-sigtran (with a more
complete SUA, M3UA and connection-oriented SCCP stack), etc.
But there are too little people working on this, and those people are
mostly coming from one particular corner, while most of the [commercial]
users do not contribute the way you would expect them to contribute in
collaborative FOSS projects. You can argue that most people in the
Linux world also don't contribute, but then the large commercial
beneficiaries (like the chipset and hardware makers) mostly do, as are
the large commercial users.
All in all, I have the feeling that Osmocom is as important as it
ever was, but it's not grown up yet to really walk on its own feet. It
may be able to crawl, though ;)
So for now, don't panic. I'm not suffering from burn-out, mid-life
crisis and I don't plan on any big changes of where I put my energy: It
will continue to be Osmocom. But I also think we have to have a more
open discussion with everyone on how to move beyond the current
situation. There's no point in staying quiet about it, or to claim that
everything is fine the way it is. We need more commitment. Not from
the people already actively involved, but from those who are not [yet].
If that doesn't happen in the next let's say 1-2 years, I think it's
fair that I might seriously re-consider in which field and in which way
I'd like to dedicate my [I would think considerable] productive energy and
I'm just on my way back from the Telecom Security Day 2017
<https://www.troopers.de/troopers17/telco-sec-day/>, which is an
invitation-only event about telecom security issues hosted by ERNW
back-to-back with their Troopers 2017 <https://www.troopers.de/troopers17/>
I've been presenting at TelcoSecDay in previous years and hence was
again invited to join (as attendee). The event has really gained quite
some traction. Where early on you could find lots of IT security /
hacker crowds, the number of participants from the operator (and to
smaller extent also equipment maker) industry has been growing.
The quality of talks was great, and I enjoyed meeting various familiar
faces. It's just a pity that it's only a single day - plus I had to
head back to Berlin still today so I had to skip the dinner + social
When attending events like this, and seeing the interesting hacks that
people are working on, it pains me a bit that I haven't really been
doing much security work in recent years. netfilter/iptables was at
least somewhat security related. My work on OpenPCD / librfid was
clearly RFID security oriented, as was the work on airprobe,
OsmocomTETRA, or even the EasyCard payment system hack
I have the same feeling when attending Linux kernel development related
events. I have very fond memories of working in both fields, and it was
a lot of fun. Also, to be honest, I believe that the work in Linux
kernel land and the general IT security research was/is appreciated much
more than the endless months and years I'm now spending my time with
improving and extending the Osmocom cellular infrastructure stack.
Beyond the appreciation, it's also the fact that both the IT security
and the Linux kernel communities are much larger. There are more
people to learn from and learn with, to engage in discussions and
ping-pong ideas. In Osmocom, the community is too small (and I have the
feeling, it's actually shrinking), and in many areas it rather seems
like I am the "ultimate resource" to ask, whether about 3GPP specs or
about Osmocom code structure. What I'm missing is the feeling of being
part of a bigger community. So in essence, my current role in the "Open
Source Cellular" corner can be a very lonely one.
But hey, I don't want to sound more depressed than I am, this was
supposed to be a post about TelcoSecDay. It just happens that attending
IT Security and/or Linux Kernel events makes me somewhat gloomy for the
Meanwhile, if you have some interesting projcets/ideas at the border
between cellular protocols/systems and security, I'd of course love to
hear if there's some way to get my hands dirty in that area again :)
As we can read in recent news, VMware has become a gold member of the
Linux foundation. That causes - to say the least - very mixed feelings to me.
One thing to keep in mind: The Linux Foundation is an industry
association, it exists to act in the joint interest of it's paying
members. It is not a charity, and it does not act for the public good.
I know and respect that, while some people sometimes appear to be
confused about its function.
However, allowing an entity like VMware to join, despite their many
years long disrespect for the most basic principles of the FOSS
Community (such as: Following the GPL and its copyleft principle),
really is hard to understand and accept.
I wouldn't have any issue if VMware would (prior to joining LF) have
said: Ok, we had some bad policies in the past, but now we fully comply
with the license of the Linux kernel, and we release all
derivative/collective works in source code. This would be a positive
spin: Acknowledge past issues, resolve the issues, become clean and then
publicly underlining your support of Linux by (among other things)
joining the Linux Foundation. I'm not one to hold grudges against
people who accept their past mistakes, fix the presence and then move
on. But no, they haven't fixed any issues.
They are having one of the worst track records in terms of intentional
GPL compliance issues for many years, showing outright disrespect for Linux,
the GPL and ultimately the rights of the Linux developers, not resolving
those issues and at the same time joining the Linux Foundation? What
kind of message sends that?
It sends the following messages:
- you can abuse Linux, the GPL and copyleft while still being accepted
amidst the Linux Foundation Members
- it means the Linux Foundations has no ethical concerns whatsoever
about accepting such entities without previously asking them to become
- it also means that VMware has still not understood that Linux and FOSS
is about your actions, particularly the kind of choices you make how
to technically work with the community, and not against it.
So all in all, I think this move has seriously damaged the image of both
entities involved. I wouldn't have expected different of VMware, but I
would have hoped the Linux Foundation had some form of standards as to
which entities they permit amongst their ranks. I guess I was being
overly naive :(
It's a slap in the face of every developer who writes code not because
he gets paid, but because it is rewarding to know that copyleft will
continue to ensure the freedom of related code.
|UPDATE (March 8, 2017):|
| ||I was mistaken in my original post in that VMware didn't just join,
but was a Linux Foundation member already before, it is "just" their
upgrade from silver to gold that made the news recently. I stand
corrected. Still doesn't make it any better that the are involved
inside LF while engaging in stepping over the lines of license
|UPDATE2 (March 8, 2017):|
| ||As some people pointed out, there is no verdict against VMware. Yes,
that's true. But the mere fact that they rather distribute derivative
works of GPL licensed software and take this to court with an armada
of lawyers (instead of simply complying with the license like everyone
else) is sad enough. By the time there will be a final verdict, the
product is EOL. That's probably their strategy to begin with :/
I always though I understood UMTS AKA (authentication and key
agreement), including the re-synchronization procedure. It's been years
since I wrote tools like osmo-sim-auth which you can use to
perform UMTS AKA with a SIM card inserted into a PC reader, i.e.
simulate what happens between the AUC (authentication center) in a
network and the USIM card.
However, it is only now as the sysmocom team works on 3G support of the
dedicated OsmoHLR (outside of
OsmoNITB!), that I seem to understand all the nasty little details.
I always thought for re-synchronization it is sufficient to simply
increment the SQN (sequence number). It turns out, it isn't as there is
a MSB-portion called SEQ and a lower-bit portion called IND, used for
some fancy array indexing scheme of buckets of highest-used-SEQ within
that IND bucket.
If you're interested in all the dirty details and associated spec
references (the always hide the important parts in some Annex) see the
discussion between Neels and me in Osmocom redmine issue 1965.
For those of you who don't know what the tinkerphones/OpenPhoenux GTA04 is: It is a
'professional hobbyist' hardware project (with at least public
schematics, even if not open hardware in the sense that editable
schematics and PCB design files are published) creating updated
mainboards that can be used to upgrade Openmoko phones. They fit into
the same enclosure and can use the same display/speaker/microphone.
What the GTA04 guys have been doing for many years is close to a miracle
anyway: Trying to build a modern-day smartphone in low quantities,
using off-the-shelf components available in those low quantities, and
without a large company with its associated financial backing.
Smartphones are complex because they are highly integrated devices. A
seemingly unlimited amount of components is squeezed in the tiniest
form-factors. This leads to complex circuit boards with many layers
that take a lot of effort to design, and are expensive to build in low
quantities. The fine-pitch components mandated by the integration
density is another issue.
Building the original GTA01 (Neo1937) and GTA02 (FreeRunner) devices at
Openmoko, Inc. must seem like a piece of cake compared to what the GTA04
guys are up to. We had a team of engineers that were familiar at last
with feature phone design before, and we had the backing of a consumer
electronics company with all its manufacturing resources and expertise.
Nevertheless, a small group of people around Dr. Nikolaus Schaller has
been pushing the limits of what you can do in a small for fun
project, and the have my utmost respect. Well done!
Unfortunately, there are bad news. Manufacturing of their latest
generation of phones (GTA04A5) has been stopped due to massive soldering
problems with the TI OMAP3 package-on-package (PoP).
Those PoPs are basically "RAM chip soldered onto the CPU, and the stack
of both soldered to the PCB". This is used to save PCB footprint and to
avoid having to route tons of extra (sensitive, matched) traces between
the SDRAM and the CPU.
According to the mailing list posts, it seems to be incredibly difficult
to solder the PoP stack due to the way TI has designed the packaging of
the DM3730. If you want more gory details, see
and yet another post.
It is very sad to see that what appears to be bad design choices at TI
are going to bring the GTA04 project to a halt. The financial hit by
having only 33% yield is already more than the small community can take,
let alone unused parts that are now in stock or even thinking about
further experiments related to the manufacturability of those chips.
If there's anyone with hands-on manufacturing experience on the DM3730
(or similar) TI PoP reading this: Please reach out to the GTA04 guys and
see if there's anything that can be done to help them.
|UPDATE (March 8, 2017):|
| ||In an earlier post I was asserting that the GTA04 is open hardware
(which I actually believed up to that point) until some readers have
pointed out to me that it isn't. It's sad it isn't, but still it has
The recent Amazon S3 outage should make a strong argument that centralized services have severe issues, technically but from a business point of view as well(you don’t own the destiny of your own product!) and I whole heartily agree with “There is no cloud, it’s only someone else’s computer”.
Still from time to time I like to see beyond my own nose (and I prefer the German version of that proverb!) and the current exploration involves ReactJS (which I like), Tensorflow (which I don’t have enough time for) and generally looking at Docker/Mesos/Kubernetes to manage services, zero downtime rolling updates. I have browsed and read the documentation over the last year, like the concepts (services, replication controller, pods, agents, masters), planned how to use it but because it doesn’t support SCTP never looked into actually using it.
Microsoft Azure has the Azure Container Services and since end of February it is possible to create Kubernetes clusters. This can be done using the v2 of the Azure CLI or through the portal. I finally decided to learn some new tricks.
Azure asks for a clientId and password and I entered garbage and hoped the necessary accounts would be created. It turns out that the portal is not creating it and also not doing a sanity check of these credentials and second when booting the master it will not properly start. The Microsoft support was very efficient and quick to point that out. I wish the portal would make a sanity check though. So make sure to create a principal first and use it correctly. I ended up creating it on the CLI.
I re-created the cluster and executed kubectl get nodes. It started to look better but one agent was missing from the list of nodes. After logging in I noticed that kubelet was not running. Trying to start it by hand shows that docker.service is missing. Now why it is missing is probably for Microsoft engineering to figure out but the Microsoft support gave me:
sudo rm -rf /var/lib/cloud/instances
sudo cloud-init -d init
sudo cloud-init -d modules -m config
sudo cloud-init -d modules -m final
sudo systemctl restart kubelet
After these commands my system would have a docker.service, kubelet would start and it will be listed as a node. Commands like kubectl expose are well integrated and use a public IPv4 address that is different from the one used for ssh/management. So all in all it was quite easy to get a cluster up and I am sure that some of the hick-ups will be fixed…
In May 2016 we got the GTP-U tunnel encapsulation/decapsulation
module developed by Pablo Neira, Andreas Schultz and myself merged into
the 4.8.0 mainline kernel.
During the second half of 2016, the code basically stayed untouched. In
early 2017, several patch series of (at least) three authors have been
published on the netdev mailing list for review and merge.
This poses the very valid question on how do we test those (sometimes
quite intrusive) changes. Setting up a complete cellular network with
either GPRS/EGPRS or even UMTS/HSPA is possible using OsmoSGSN and
related Osmocom components. But it's of course a luxury that not many
Linux kernel networking hackers have, as it involves the availability of
a supported GSM BTS or UMTS hNodeB. And even if that is available,
there's still the issue of having a spectrum license, or a wired setup
with coaxial cable.
So as part of the recent discussions on netdev, I tested and described a
minimal test setup using libgtpnl, OpenGGSN and sgsnemu.
This setup will start a mobile station + SGSN emulator inside a Linux
network namespace, which talks GTP-C to OpenGGSN on the host, as well as
GTP-U to the Linux kernel GTP-U implementation.
In case you're interested, feel free to check the following wiki page:
This is of course just for manual testing, and for functional (not
performance) testing only. It would be great if somebody would pick up
on my recent mail containing some suggestions about an automatic
regression testing setup for the kernel GTP-U code. I have way
too many spare-time projects in desperate need of some attention to work
on this myself. And unfortunately, none of the telecom operators (who
are the ones benefiting most from a Free Software accelerated GTP-U
implementation) seems to be interested in at least co-funding or
otherwise contributing to this effort :/
Keeping up my yearly blogging cadence, it’s about time I wrote to let people know what I’ve been up to for the last year or so at Mozilla. People keeping up would have heard of the sad news regarding the Connected Devices team here. While I’m sad for my colleagues and quite disappointed in how this transition period has been handled as a whole, thankfully this hasn’t adversely affected the Vaani project. We recently moved to the Emerging Technologies team and have refocused on the technical side of things, a side that I think most would agree is far more interesting, and also far more suited to Mozilla and our core competence.
So, out with Project Vaani, and in with Project DeepSpeech (name will likely change…) – Project DeepSpeech is a machine learning speech-to-text engine based on the Baidu Deep Speech research paper. We use a particular layer configuration and initial parameters to train a neural network to translate from processed audio data to English text. You can see roughly how we’re progressing with that here. We’re aiming for a 10% Word Error Rate (WER) on English speech at the moment.
You may ask, why bother? Google and others provide state-of-the-art speech-to-text in multiple languages, and in many cases you can use it for free. There are multiple problems with existing solutions, however. First and foremost, most are not open-source/free software (at least none that could rival the error rate of Google). Secondly, you cannot use these solutions offline. Third, you cannot use these solutions for free in a commercial product. The reason a viable free software alternative hasn’t arisen is mostly down to the cost and restrictions around training data. This makes the project a great fit for Mozilla as not only can we use some of our resources to overcome those costs, but we can also use the power of our community and our expertise in open source to provide access to training data that can be used openly. We’re tackling this issue from multiple sides, some of which you should start hearing about Real Soon Now™.
The whole team has made contributions to the main code. In particular, I’ve been concentrating on exporting our models and writing clients so that the trained model can be used in a generic fashion. This lets us test and demo the project more easily, and also provides a lower barrier for entry for people that want to try out the project and perhaps make contributions. One of the great advantages of using TensorFlow is how relatively easy it makes it to both understand and change the make-up of the network. On the other hand, one of the great disadvantages of TensorFlow is that it’s an absolute beast to build and integrates very poorly with other open-source software projects. I’ve been trying to overcome this by writing straight-forward documentation, and hopefully in the future we’ll be able to distribute binaries and trained models for multiple platforms.
We’re still at a fairly early stage at the moment, which means there are many ways to get involved if you feel so inclined. The first thing to do, in any case, is to just check out the project and get it working. There are instructions provided in READMEs to get it going, and fairly extensive instructions on the TensorFlow site on installing TensorFlow. It can take a while to install all the dependencies correctly, but at least you only have to do it once! Once you have it installed, there are a number of scripts for training different models. You’ll need a powerful GPU(s) with CUDA support (think GTX 1080 or Titan X), a lot of disk space and a lot of time to train with the larger datasets. You can, however, limit the number of samples, or use the single-sample dataset (LDC93S1) to test simple code changes or behaviour.
One of the fairly intractable problems about machine learning speech recognition (and machine learning in general) is that you need lots of CPU/GPU time to do training. This becomes a problem when there are so many initial variables to tweak that can have dramatic effects on the outcome. If you have the resources, this is an area that you can very easily help with. What kind of results do you get when you tweak dropout slightly? Or layer sizes? Or distributions? What about when you add or remove layers? We have fairly powerful hardware at our disposal, and we still don’t have conclusive results about the affects of many of the initial variables. Any testing is appreciated! The Deep Speech 2 paper is a great place to start for ideas if you’re already experienced in this field. Note that we already have a work-in-progress branch implementing some of these ideas.
Let’s say you don’t have those resources (and very few do), what else can you do? Well, you can still test changes on the LDC93S1 dataset, which consists of a single sample. You won’t be able to effectively tweak initial parameters (as unsurprisingly, a dataset of a single sample does not represent the behaviour of a dataset with many thousands of samples), but you will be able to test optimisations. For example, we’re experimenting with model quantisation, which will likely be one of multiple optimisations necessary to make trained models usable on mobile platforms. It doesn’t particularly matter how effective the model is, as long as it produces consistent results before and after quantisation. Any optimisation that can be made to reduce the size or the processor requirement of training and using the model is very valuable. Even small optimisations can save lots of time when you start talking about days worth of training.
Our clients are also in a fairly early state, and this is another place where contribution doesn’t require expensive hardware. We have two clients at the moment. One written in Python that takes advantage of TensorFlow serving, and a second that uses TensorFlow’s native C++ API. This second client is the beginnings of what we hope to be able to run on embedded hardware, but it’s very early days right now.
Imagine a future where state-of-the-art speech-to-text is available, for free (in cost and liberty), on even low-powered devices. It’s already looking like speech is going to be the next frontier of human-computer interaction, and currently it’s a space completely tied up by entities like Google, Amazon, Microsoft and IBM. Putting this power into everyone’s hands could be hugely transformative, and it’s great to be working towards this goal, even in a relatively modest capacity. This is the vision, and I look forward to helping make it a reality.
I've recently attended a seminar that (among other topics) also covered
RF interference hunting. The speaker was talking about various
real-world cases of RF interference and illustrating them in detail.
Of course everyone who has any interest in RF or cellular will know
about fundamental issues of radio frequency interference. To the
biggest part, you have
- cells of the same operator interfering with each other due to too
frequent frequency re-use, adjacent channel interference, etc.
- cells of different operators interfering with each other due to
intermodulation products and the like
- cells interfering with cable TV, terrestrial TV
- DECT interfering with cells
- cells or microwave links interfering with SAT-TV reception
- all types of general EMC problems
But what the speaker of this seminar covered was actually a cellular
base-station being re-broadcast all over Europe via a commercial
It is a well-known fact that most satellites in the sky are basically
just "bent pipes", i.e. they consist of a RF receiver on one frequency,
a mixer to shift the frequency, and a power amplifier. So basically
whatever is sent up on one frequency to the satellite gets
re-transmitted back down to earth on another frequency. This is abused
by "satellite hijacking" or "transponder hijacking" and has been covered
for decades in various publications.
Ok, but how does cellular relate to this? Well, apparently some people
are running VSAT terminals (bi-directional satellite terminals) with
improperly shielded or broken cables/connectors. In that case, the RF
emitted from a nearby cellular base station leaks into that cable, and
will get amplified + up-converted by the block up-converter of that VSAT
The bent-pipe satellite subsequently picks this signal up and
re-transmits it all over its coverage area!
I've tried to find some public documents about this, an there's
surprisingly little public information about this phenomenon.
However, I could find a slide set from SES, presented at a
Satellite Interference Reduction Group: Identifying Rebroadcast (GSM)
It describes a surprisingly manual and low-tech approach at hunting down
the source of the interference by using an old nokia net-monitor phone
to display the MCC/MNC/LAC/CID of the cell. Even in 2011 there were
already open source projects such as airprobe that could have done the
job based on sampled IF data. And I'm not even starting to consider
It should be relatively simple to have a SDR that you can tune to a
given satellite transponder, and which then would look for any
GSM/UMTS/LTE carrier within its spectrum and dump their identities in a
fully automatic way.
But then, maybe it really doesn't happen all that often after all to
rectify such a development...
In the good old days ever since the late 1980ies - and a surprising
amount even still today - telecom signaling traffic is still carried
over circuit-switched SS7 with its TDM lines as physical layer, and not
an IP/Ethernet based transport.
When Holger first created OsmoBSC, the BSC-only version of OpenBSC some
7-8 years ago, he needed to implement a minimal subset of SCCP wrapped
in TCP called SCCP Lite. This was due to the simple fact that the MSC
to which it should operate implemented this non-standard protocol
stacking that was developed + deployed before the IETF SIGTRAN WG
specified M3UA or SUA came around. But even after those were specified
in 2004, the 3GPP didn't specify how to carry A over IP in a standard
way until the end of 2008, when a first A interface over IP study
As time passese, more modern MSCs of course still implement classic
circuit-switched SS7, but appear to have dropped SCCPlite in favor of
real AoIP as specified by 3GPP meanwhile. So it's time to add this to
the osmocom universe and OsmoBSC.
A couple of years ago (2010-2013) implemented both classic SS7
(MTP2/MTP3/SCCP) as well as SIGTRAN stackings (M2PA/M2UA/M3UA/SUA in
Erlang. The result has been used in some production deployments, but
only with a relatively limited feature set. Unfortunately, this code
has nto received any contributions in the time since, and I have to say
that as an open source community project, it has failed. Also, while
Erlang might be fine for core network equipment, running it on a BSC
really is overkill. Keep in miond that we often run OpenBSC on
really small ARM926EJS based embedded systems, much more resource
constrained than any single smartphone during the late decade.
In the meantime (2015/2016) we also implemented some minimal SUA support
for interfacing with UMTS femto/small cells via Iuh (see OsmoHNBGW).
So in order to proceed to implement the required
SCCP-over-M3UA-over-SCTP stacking, I originally thought well, take
Holgers old SCCP code, remove it from the IPA multiplex below, stack it
on top of a new M3UA codebase that is copied partially from SUA.
However, this falls short of the goals in several ways:
- The application shouldn't care whether it runs on top of SUA or SCCP,
it should use a unified interface towards the SCCP Provider.
OsmoHNBGW and the SUA code already introduce such an interface baed on
the SCCP-User-SAP implemented using Osmocom primitives (osmo_prim).
However, the old OsmoBSC/SCCPlite code doesn't have such abstraction.
- The code should be modular and reusable for other SIGTRAN stackings
as required in the future
So I found myself sketching out what needs to be done and I ended up
pretty much with a re-implementation of large parts. Not quite fun, but
definitely worth it.
The strategy is:
And then finally stack all those bits on top of each other, rendering a
fairly clean and modern implementation that can be used with the IuCS of
the virtually unmodified OsmmoHNBGW, OsmoCSCN and OsmoSGSN for testing.
Next steps in the direction of the AoIP are:
- Implementation of the MTP-SAP based on the IPA transport
- Binding the new SCCP code on top of that
- Converting OsmoBSC code base to use the SCCP-User-SAP for its
From that point onwards, OsmoBSC doesn't care anymore whether it
transports the BSSAP/BSSMAP messages of the A interface over
SCCP/IPA/TCP/IP (SCCPlite) SCCP/M3UA/SCTP/IP (3GPP AoIP), or even
something like SUA/SCTP/IP.
However, the 3GPP AoIP specs (unlike SCCPlite) actually modify the
BSSAP/BSSMAP payload. Rather than using Circuit Identifier Codes and
then mapping the CICs to UDP ports based on some secret conventions,
they actually encapsulate the IP address and UDP port information for
the RTP streams. This is of course the cleaner and more flexible
approach, but it means we'll have to do some further changes inside the
actual BSC code to accommodate this.
When implementing any kind of communication protocol, one always dreams
of some existing test suite that one can simply run against the
implementation to check if it performs correct in at least those use
cases that matter to the given application.
Of course in the real world, there rarely are protocols where this is
true. If test specifications exist at all, they are often just very
abstract texts for human consumption that you as the reader should
For some (by far not all) of the protocols found in cellular networks,
every so often I have seen some formal/abstract machine-parseable test
specifications. Sometimes it was TTCN-2, and sometimes TTCN-3.
If you haven't heard about TTCN-3, it is basically a way to create
functional tests in an abstract description (textual + graphical), and
then compile that into an actual executable tests suite that you can run
against the implementation under test.
However, when I last did some research into this several years ago, I
couldn't find any Free / Open Source tools to actually use those
formally specified test suites. This is not a big surprise, as even
much more fundamental tools for many telecom protocols are missing, such
as good/complete ASN.1 compilers, or even CSN.1 compilers.
To my big surprise I now discovered that Ericsson had released their
(formerly internal) TITAN TTCN3 Toolset
as Free / Open Source Software under EPL 1.0. The project is even part
of the Eclipse Foundation. Now I'm certainly not a friend of Java or
Eclipse by all means, but well, for running tests I'd certainly not
The project also doesn't seem like it was a one-time code-drop but seems
very active with many repositories on gitub. For example for the core
module, titan.core shows
plenty of activity on an almost daily basis. Also, binary releases for
a variety of distributions are made available. They
even have a video showing the installation ;)
If you're curious about TTCN-3 and TITAN, Ericsson also have made
available a great 200+ pages slide set about TTCN-3 and TITAN.
I haven't yet had time to play with it, but it definitely is rather high
on my TODO list to try.
ETSI provides a couple of test suites in TTCN-3 for protocols like
DIAMETER, GTP2-C, DMR, IPv6, S1AP, LTE-NAS, 6LoWPAN, SIP, and others at
http://forge.etsi.org/websvn/ (It's also the first time I've seen that
ETSI has a SVN server. Everyone else is using git these days, but yes,
revision control systems rather than periodic ZIP files is definitely a
big progress. They should do that for their reference codecs and ASN.1
I'm not sure once I'll get around to it. Sadly, there is no TTCN-3 for
SCCP, SUA, M3UA or any SIGTRAN related stuff, otherwise I would want to
try it right away. But it definitely seems like a very interesting
technology (and tool).
Last weekend I had the pleasure of attending FOSDEM 2017. For many years, it is probably the most
exciting event exclusively on Free Software to attend every year.
My personal highlights (next to meeting plenty of old and new friends)
in terms of the talks were:
I was attending but not so excited by Georg Greve's OpenPOWER talk. It was a
great talk, and it is an important topic, but the engineer in me would
have hoped for some actual beefy technical stuff. But well, I was just
not the right audience. I had heard about OpenPOWER quite some time ago
and have been following it from a distance.
The LoRaWAN talk
couldn't have been any less technical, despite stating technical,
political and cultural in the topic. But then, well, just recently
33C3 had the most exciting LoRa PHY Reverse Engineering Talk by Matt
Other talks whose recordings I still want to watch one of these days:
I'm very happy that in 2017, we will have the first ever technical
conference on the Osmocom cellular infrastructure projects.
For many years, we have had a small, invitation only event by Osmocom
developers for Osmocom developers called OsmoDevCon. This was fine for
the early years of Osmocom, but during the last few years it became
apparent that we also need a public event for our many users. Those
range from commercial cellular operators to community based efforts like
Rhizomatica, and of course include the many
research/lab type users with whom we started.
So now we'll have the public OsmoCon on April 21st, back-to-back with
the invitation-only OsmoDevcon from April 22nd through 23rd.
I'm hoping we can bring together a representative sample of our user
base at OsmoCon 2017 in April. Looking forward to meet you all. I hope
you're also curious to hear more from other users, and of course the
A few days ago, Autodesk has announecd
that the popular EAGLE electronics design automation (EDA) software is
moving to a subscription based model.
When previously you paid once for a license and could use that
version/license as long as you wanted, there now is a monthly
subscription fee. Once you stop paying, you loose the right to use the
software. Welcome to the brave new world.
I have remotely observed this subscription model as a general trend in
the proprietary software universe. So far it hasn't affected me at all,
as the only two proprietary applications I use on a regular basis
during the last decade are IDA and EAGLE.
I already have ethical issues with using non-free software, but those
two cases have been the exceptions, in order to get to the productivity
required by the job. While I can somehow convince my consciousness in
those two cases that it's OK - using software under a subscription model is
completely out of the question, period. Not only would I end up paying
for the rest of my professional career in order to be able to open and
maintain old design files, but I would also have to accept software that
"calls home" and has "remote kill" features. This is clearly not
something I would ever want to use on any of my computers. Also, I
don't want software to be associated with any account, and it's not the
bloody business of the software maker to know when and where I use my
For me - and I hope for many, many other EAGLE users - this move is
utterly unacceptable and certainly marks the end of any business between
the EAGLE makers and myself and/or my companies. I will happily use
my current "old-style" EAGLE 7.x licenses for the near future, and theS
see what kind of improvements I would need to contribute to KiCAD or
other FOSS EDA software in order to eventually migrate to those.
As expected, this doesn't only upset me, but many other customers, some
of whom have been loyal to using EAGLE for many years if not decades,
back to the DOS version. This is reflected by some media reports (like
this one at hackaday
or user posts at element14.com or eaglecentral.ca
who are similarly critical of this move.
Rest in Peace, EAGLE. I hope Autodesk gets what they deserve: A new
influx of migrations away from EAGLE into the direction of Open Source
EDA software like KiCAD.
In fact, the more I think about it, I'm actually very much inclined to
work on good FOSS migration tools / converters - not only for my own
use, but to help more people move away from EAGLE. It's not that I
don't have enough projects at my hand already, but at least I'm
motivated to do something about this betrayal by Autodesk. Let's see
what (if any) will come out of this.
So let's see it that way: What Autodesk is doing is raising the level
off pain of using EAGLE so high that more people will use and contribute
FOSS EDA software. And that is actually a good thing!
I've just had the pleasure of attending all four days of 33C3 and have returned
home with somewhat mixed feelings.
I've been a regular visitor and speaker at CCC events since 15C3 in
1998, which among other things
means I'm an old man now. But I digress ;)
The event has come extremely far in those years. And to be honest, I
struggle with the size. Back then, it was a meeting of like-minded
hackers. You had the feeling that you know a significant portion of the
attendees, and it was easy to connect to fellow hackers.
These days, both the number of attendees and the size of the event make
you feel much rather that you're in general public, rather than at some
meeting of fellow hackers. Yes, it is good to see that more people are
interested in what the CCC (and the selected speakers) have to say, but
somehow it comes at the price that I (and I suspect other old-timers)
feel less at home. It feels too much like various other technology
One aspect creating a certain feeling of estrangement is also the venue
itself. There are an incredible number of rooms, with a labyrinth of
hallways, stairs, lobbies, etc. The size of the venue simply makes it
impossible to simply _accidentally_ running into all of your fellow
hackers and friends. If I want to meet somebody, I have to make an
explicit appointment. That is an option that exits most of the rest of
the year, too.
While fefe is happy about the many small children attending
the event, to me this seems
somewhat alien and possibly inappropriate. I guess from teenage years
onward it certainly makes sense, as they can follow the talks and
participate in the workshop. But below that age?
The range of topics covered at the event also becomes wider, at least I
feel that way. Topics like IT security, data protection, privacy,
intelligence/espionage and learning about technology have always been
present during all those years. But these days we have bloggers sitting
on stage and talking about bottles of wine (seriously?).
Contrary to many, I also really don't get the excitement about shows
like 'Methodisch Inkorrekt'. Seems to me like mainstream
compatible entertainment in the spirit of the 1990ies Knoff Hoff Show without much
potential to make the audience want to dig deeper into (information)
Yesterday, together with Holger 'zecke' Freyther, I co-presented at 33C3 about
Dissectiong modern (3G/4G) cellular modems.
This presentation covers some of our recent explorations into a specific
type of 3G/4G cellular modems, which next to the regular modem/baseband
processor also contain a Cortex-A5 core that (unexpectedly) runs Linux.
We want to use such modems for building self-contained M2M devices that
run the entire application inside the modem itself, without any external
needs except electrical power, SIM card and antenna.
Next to that, they also pose an ideal platform for testing the Osmocom
network-side projects for running GSM, GPRS, EDGE, UMTS and HSPA
You can find the Slides
and the Video recordings
in case you're interested in more details about our work.
The results of our reverse engineering can be found in the wiki at
http://osmocom.org/projects/quectel-modems/wiki together with links to
the various git repositories containing related tools.
As with all the many projects that I happen to end up doing, it would be
great to get more people contributing to them. If you're interested in
cellular technology and want to help out, feel free to register at the
osmocom.org site and start adding/updating/correcting information to the
You can e.g. help by
- playing with the modem and documenting your findings
- reviewing the source code released by Qualcomm + Quectel and
documenting your findings
- help us to create a working OE build with our own kernel and rootfs
images as well as opkg package feeds for the modems
- help reverse engineering DIAG and QMI protocols as well as the open
source programs to interact with them