Thursday afternoon… last thing I’d like to see is a mail with similar subject from the CTO of the company.

$ docker login -u $CONTAINER_REGISTRY_USERNAME -p $CONTAINER_REGISTRY_PASSWORD $CONTAINER_REGISTRY
Error response from daemon: login attempt to https://registry.XXXXXX/v2/ failed with status: 401 Unauthorized

For various reasons we run our own private container registry. Needless to say, without it all development is blocked. Worse, still if a running pod needs to be started on another node, the new node MAY not contain the image in it’s local cache so a pod migration(a normal event in kubernetes) can actually result in a service outage. I know it’s a single point of failure, have a ticket in the backlog about it.

The setup is quite straightforward, Registry is deployed in k8s behind an LB, uses s3 for storage and Keycloak for AA. It’s been worry-free for the past year or so, apart from some semi-periodic clean-up of old cruft.

Looking at the logs I saw a lot of errors:

time="2025-04-04T09:32:55.822077511Z" level=error msg="traces export: Post \"https://localhost:4318/v1/traces\": dial tcp [::1]:4318: connect: connection refused" environment=development go.version=go1.23.7 instance.id=1df5abf0-2cc7-4268-9d28-b3f4a046ce87 service=registry version=3.0.0

So why the hell is registry all of a sudden trying to reach localhost:4318? I know, by accident, that such a request is actually the service trying to reach out to an Otel(OpenTelemetry) collector but why would that cause an outage?

Well, it didn’t. Hidden among the thousand similar errors there was also another line:

time="2025-04-04T09:30:53.436663295Z" level=info msg="failed to verify token: token signed by untrusted key with ID: \"FML4:7GSU:IBQ4:XXX\""

Let me sink that for a moment. I got 100+ errors that Otel is unable to post it’s traces, but I get a single info message why my AA setup is not working!

Google led to this issue: INFO[0101] failed to verify token: token signed by untrusted key with ID. It seems the logic inside Registry has changed and now the Keycloak setup that’s been working for some time and that is done exactly as per Keycloack documentation needs additional configuration.

But why did it break now? Well, in the deployment I had:

kind: Deployment
metadata:
  name: registry
  ...
spec:
  template:
    spec:
      containers:
      - name: registry
        image: registry:latest

And guess what? :latest just got bumped to 3.0.0 yesterday. So it was just a matter of time until the pod was restarted for disaster to strike. For now, I’ve just changed to image: registry:2 and that seems to have fixed the issue on prod.

However, I feel kind of alienated by the registry guys because:

• They knowingly broke token auth for legacy implementations. No migration path, not even a mention besides “docker/libtrust has been replaced with go-jose/go-jose in #4096”. I for sure don’t know what go-jose is and don’t really care as long as it does not make me do more work. When a fix/pull request was proposed it was dismissed as pretty much We used to do bad things, but now we do only good ones.

• OTel traces being enabled by default is a PITA. If it’s an opt-in - fine, I’ll just not use it. If there’s a way to disable it via config file option - also somewhat OK. Luckily it’s in k8s so I can just do:

    ```
    configMapGenerator:
    - name: registry-new-config
      literals:
      - REGISTRY_LOG_LEVEL="info"
      - OTEL_TRACES_EXPORTER="none"
    ```
  

  But what if it runs on real hardware/shared VM?

Both these issues shows to me that the maintainers of registry just don’t care about their users…

Just sure to restart your Alloy instances from time to time

A Monitoring tale

I’ve been setting up a LG(T)M stack to monitor a kubernetes cluster of mine. It’s a self-built setup running on Hetzner Cloud using the excellent Kube-Hetzner project to configure the cluster itself. I’ll probably do a write-up on that setup at a later point, suffice to say that it’s really hassle-free and works great.

I have a monitoring/alerting set-up consisting of Grafana itself for visualisation, Grafana Loki for log storage, Grafana Mimir for metrics storage and Grafana Alloy to scrape logs and metrics. It’s been running for a few months, delivering logs and metrics as requested - not much as it’s a dev/demo system that does not see much usage.

Lately I’ve looked into pushing more workloads to the cluster and decided I need more metrics at pod level. Lookin at what was scraped, I identified two sets of metrics that I wanted in - specifically Kube-State-Metrics and cAdvisor. It’s a bit of a mess to set that up in Alloy (another piece to describe in a future post) and it involved lots of edit to the Alloy configuration file. At one point in time I started receiving some errors in both Mimir and Alloy logs:

ts=2025-01-29T14:12:10.555161198Z level=error msg="non-recoverable error" component_path=/ 
component_id=prometheus.remote_write.default_mimir subcomponent=rw remote_name=711a87 
url=http://mimir-nginx.monitoring.svc.cluster.local:80/api/v1/push count=707 exemplarCount=0 
err="server returned HTTP status 400 Bad Request: received a series with duplicate label name, 
label: 'node_kubernetes_io_instance_type' series: 'machine_scrape_error{beta_kubernetes_io_arch=\"amd64\",
beta_kubernetes_io_instance_type=\"cx42\", beta_kubernetes_io_os=\"linux\", csi_hetzner_cloud_location=\"hel1\",
 failure_domain_beta_kubernetes_io_regio' (err-mimir-duplicate-label-names)"

This is just a sample, I was getting lots of messages for different duplicate labels, not only node_kubernetes_io_instance_type.

Google showed up some results like This and This. However, the Quick Workaround mentioned didn’t work for me… I dug a bit deeper and these two issues didn’t really seem to be the same as mine - in both cases an err-mimir-duplicate-label-names was generated, but I was not doing any relabelling on my metrics at all.

After spending a few hours fighting with this, including disabling almost all metrics collection and still getting errors I was out of ideas.

Now a step back. I’m using Alloy in a distributed way in a daemonset, meaning each node gets it’s own Alloy pod - in my case I had 6 of them. Also, I’ve configured Alloy to do live config reloads to minimize downtime. This all meant that my Alloy pods were up for 10+ days… In an act of desperation I followed the old Level-1 Tech support advice: “Have you tried turning it off and on again?”. In my case:

kubectl rollout restart daemonset.apps/alloy -n monitoring

When the restart was complete and the Alloy WAL was flushed from previous data, there were no more errors!

What has probably happened is that during all the configuration file restarts, Alloy instances got out of sync or just had their internal state messed up. This will probably not happen during normal operation as either:

• Config file(s) are rarely changed
• Pods get restarted from time to time
• Live reload is not used

But it’s good practice to keep in mind that sometimes a good old restart is the proper solution.

Addendum: two days later

After some more experiments I have a pretty reliable way of triggering this. Configure Alloy to scrape more metrics than Mimir will accept, you’ll soon get a HTTP 429 like so:

ts=2025-01-31T21:11:59.769007025Z level=warn msg="Failed to send batch, retrying" component_path=/ 
component_id=prometheus.remote_write.default_mimir subcomponent=rw remote_name=183919 
url=http://mimir-nginx.monitoring.svc.cluster.local:80/api/v1/push 
err="server returned HTTP status 429 Too Many Requests: the request has been rejected 
because the tenant exceeded the ingestion rate limit, set to 10000 items/s with a maximum 
allowed burst of 200000. This limit is applied on the total number of samples, exemplars 
and metadata received across all distributors (err-mimir-tenant-max-ingestion-rate). 
To adjust the related per-tenant limits, configure -distributor.ingestion-rate-limit 
and -distributor.ingestion-burst-size, or contact your service administrator."

Fair enough, Adjust the limits or reduce metrics volume and the error should go away, right? Yep, but more often than not err-mimir-duplicate-label-names will start popping up, prompting a restart of Alloy.

Bonus: Singleline query Kubernetes API from within a pod with curl

For my own easy reference:

curl \
  --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt   \
  --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://kubernetes.default.svc.cluster.local/api/v1/nodes/${NODE}/proxy/pods

If you get a One or more issues occurred while trying to rotate service endpoint secrets… - create a new service connection with the same settings and do a clean pipeline run.

Just another Monday morning

- Hey Ivan, the pipeline started failing during the weekend and we urgently need it for a release

Sure I can. It’s a project I just recently inherited and only have a basic idea on the setup, all running on Azure with the code and pipelines on Azure DevOps.

Pipeline failed during Docker push stage like so:

So where are the credentials stored? Quick reasearch and in Azure DevOps it’s common to define Service connections. Opening the properties page of the SC that the pipeline used for the push showed an error message: One or more issues occurred while trying to rotate service endpoint secrets. More details: Insufficient privileges to complete the operation in Microsoft Graph.

Great… so how do I fix that? The single thing one can do to a service connection is “Edit” it. According to this article, there must be a “Verify” button - but there was none. I the same article there’s a mention to do a dummy edit (e.g. update the description field) - tested it, didn’t work.

After scratching my head for a few moments decided to replace the connection. Renamed the old one to xxx-old then created a new one using the exact same settings. Re-ran the failed stage… and it failed again.

At that point I had no other ideas how to proceed. So put in some additional logging in the pipeline and ran it again - which worked! Hurray!

Analysis/speculation

Why did the pipeline fail in the first place and why did the re-run fail as well even with the new connection in place?

I can only speculate, but probably the initial error message was due to the creator of the Service Connection (DL - the previous DevOps guy, you can see him in the first picture) not being part of the organization anymore. Probably the automation that works behind the scenes tries to use his permissions to create a Service Principal or similar - which fails.

But why did the pipeline rerun failed? Again a speculation: The service connection is referred to in the pipeline by name. During initial pipeline run setup, the runner fetches the UUID of the service connection and caches it for consecutive runs, effectively still using the old connection. A new run looked it up again and that’s why it worked.

A totally unscientific benchmark of GCC on a real project that I’m currently working.

Why?

Because switching from GCC 6 to GCC12 felt slow and I wanted to know exactly how much slower it really is.

Background

It’s a real project, a custom firmware for a Cortex-M4F MCU (Atmel AT-SAME51 to be specific). It’s developed mainly in plain old C, with a bit of C++ thrown in. Even when C++ is used, it’s mostly as C-with-classes. The specific compiler options in place are:

-Os 
-mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 
-ffunction-sections -mlong-calls 
-Waddress -Wformat-nonliteral -Wformat-security -Wformat -Winit-self -Wmissing-declarations 
-Wno-multichar -Wunreachable-code -Wwrite-strings -Wpointer-arith -Wfloat-equal -Wparentheses 
-Wunused-parameter -Wunused-variable -Wreturn-type -Wuninitialized -Wextra -Wall  
-std=gnu11 -MD -MT

Methodology

Execute the below script two times, changing PATH to point to different compiler versions.

test_compile () {
    sleep 30
    make clean
    time make -j4 | grep real
}

for i in {1..10}; do test_compile; done

Basically compile the firmware 10 times in a loop, using up to 4 parallel compiler invokations - as this is the number of cores that I have on my setup.

Raw Results

GCC 6.3.1

$ arm-none-eabi-gcc --version 
arm-none-eabi-gcc (Atmel build: 508) 6.3.1 20170620 (release) [ARM/embedded-6-branch revision 249437]

$ ./test_compile.sh | grep real
real	0m13,659s
real	0m14,090s
real	0m15,397s
real	0m14,367s
real	0m14,815s
real	0m15,611s
real	0m18,213s
real	0m15,756s
real	0m18,644s
real	0m17,026s

GCC 12.2

$ arm-none-eabi-gcc --version 
arm-none-eabi-gcc (Arm GNU Toolchain 12.2.MPACBTI-Rel1 (Build arm-12-mpacbti.34)) 12.2.1 20230214

$ ./test_compile.sh | grep real
real	0m20,078s
real	0m22,832s
real	0m19,660s
real	0m19,067s
real	0m18,097s
real	0m18,159s
real	0m19,469s
real	0m19,113s
real	0m18,993s
real	0m20,311s

Results:

Build time

Build time in seconds:

Binary size

Sizes in bytes as obtained from arm-none-eabi-size:

Conclusion

GCC 12 is around 25% slower, which is noticeable even with this small project if doing a full build. I don’t have a statistic how much of that time is spent between the compiler and linker but subjectively even building and linking only a few files do seem slower. On the plus side, GCC 12 produces a binary that’s ~600 bytes smaller - which is rarely a deal-breaker but it may be under specific circumstances.

Build an archive similar to the ones available from CIPD

Why?

Try and get Fuchsia built on a new platform.

How?

TL/DR

This is an old post. Probably a lot hast changed. But it felt like a game of cat and mouse and I did not manage to see through it all.

Long version

Setting the stage

I’ve been playing around with Fuchsia since it went public a few years ago, periodically building the latest version and taking a look at how it evolved. Then my family grew bigger and I suddenly found myself with an unlikely source of spare time - 15 minutes now, 10 - in two hours, mixed in between changing nappies and taking care of the household. The highly irregular and ultimately quite short periods meant that no ‘serious’ work can be done. At the same time, I was preparing for a conference talk on Risc-V - slides available here. And one night probably around 3AM a tought occured - Can I run Fuchsia, or at least Zircon, on Risc-V?

Google quickly turned up an answer - back when it was still called Magenta, Zircon has been ported to Risc-V - a good start, but outdated by several years. At present, Zircon is part of the Fuchsia checkout, let’s try building it using the native Fuchsia way. That’s the kind of task that perfectly matches my sketchy free time - make a change, kick off a build, come back later, fix new failures and retry. And I was down the rabbit hole, only to find myself trying to build Clang the way Google does.

Context

When building Fuchsia some packages are downloaded as prebuilt archives. This is done for convenience, to cut down on build resource requirements (both time and space) as well as to reuse what’s already done in another place. To do so, it relies on a tool called CIPD - short from Chrome Infrastructure Package Deployment. Although the tool is open-source. CIPD officially supports only x64 and arm64 and obviously it cannot be of any help in my quest to Risc-V compilation. Which is fine, I’m used to cross-compilation ever since I was in the dev boards business - which is quite a long time ago.

What’s annoying, however, is that Fuchsia build systems makes it seemingly easy to specify a new target architecture, but it quickly escalates once you understand that quite a lot of packages are actually prebuilt and often packaged in a particular way. Maybe I’ll be doing a separate write-up with more details, but long story short - after a few days of building toolchains and messing around, I decided to tackle the problem at it’s root and try doing what Fuchsia expects - build the packages using CIPD.

Details

While basic usage (e.g. downloading) of CIPD packages is unrestricted, there is a profound lack of information on how actually CIPD works and, more importantly, how packages are built. There is documentation, but it is usually centered at specific tools, is outdated or assumes one is a googler - which I am not.

What intially led to some confusion was what exactly CIPD does and what it does not. And building a package CIPD does not do. Rather, there is a separate set of tools, living under the Chromium repository that seem to be doing the building. Studiying the documentation, there should be a set of ‘recipes’ specifying how to build a package - which sounds reasonable. Now, where would one find the recipe for clang - and the other Fuchsia-required packages? Maybe they live in lucy-recipes? Nah, that would’ve been too obvious. Back to reading documentation, duckduckgoing/googling and when that did not work, grep-ing the fuchsia and chromium source code. And there was a lead:

ivan@balha:~/fuchsia$ rg recipes
...
tools/integration/fint/README.md
244:into the [recipes][fuchsia-recipes] repository so recipes can construct
275:[fuchsia-recipes]: https://fuchsia.googlesource.com/infra/recipes 

Aha! A quick git clone https://fuchsia.googlesource.com/infra/recipes/ && find ./ -name "*clang* turned out a ./recipes/contrib/clang_toolchain.py. Studying the contents I was sure I’m on the right track - there are options to turn on/off LTO, use LLD on OSX and a few other Clang/LLVM-specific ones. Time to kick off this build!

ivan@balha:~/Work/fuchsia-infra/recipes$ ./recipes.py run contrib/clang_toolchain enable_lto=thin

At first, I experienced a failure related to a missing git directory, fixed by running

ivan@balha:~/Work/fuchsia-infra/recipes$ mkdir ~/Work/fuchsia-infra/recipes/.recipe_deps/recipe_engine/workdir/cache/git

Next, the recipe assumes goma is available and complains with authentication/configuration failures. No command-line switch seems to exist, thus a quick manual patch got the job done:

ivan@balha:~/Work/fuchsia-infra/recipes$ git diff recipes/contrib/clang_toolchain.py
diff --git a/recipes/contrib/clang_toolchain.py b/recipes/contrib/clang_toolchain.py
index 7d68cf787..7df0d2fc4 100644
--- a/recipes/contrib/clang_toolchain.py
+++ b/recipes/contrib/clang_toolchain.py
@@ -268,16 +268,17 @@ def RunSteps(
     use_inliner_model,
     builders,
 ):
-    use_goma = (
-        not api.platform.arch == "arm" and api.platform.bits == 64
-    ) and not api.platform.is_win
+    #use_goma = (
+    #    not api.platform.arch == "arm" and api.platform.bits == 64
+    #) and not api.platform.is_win
+    use_goma = False
     if use_goma:
         api.goma.ensure()
         ninja_jobs = api.goma.jobs
         goma_context = api.goma.build_with_goma()
     else:
         ninja_jobs = api.platform.cpu_count
-        goma_context = contextlib.nullcontext()
+        goma_context = None

     # TODO: builders would ideally set this explicitly

Running the recipe command again resulted in 5 hours of compilation and testing (on a slow 4C/4T Ryzen 3 machine) which worked OK, only for the process to fail with a:

[E2021-11-09T01:33:50.197624+02:00 3124 0 annotate.go:273] original error: interactive login is required
#4 runtime/asm_amd64.s:1371 - runtime.goexit()
cas: failed to create cas client: failed to get PerRPCCredentials: interactive login is required

Never heard of CAS before. Googling does not produce meaningful results at first. Running the executable gives:

ivan@balha:~/Work/zmeiresearch.github.io/blog$ /mnt/phys2/home/ivan/Work/fuchsia-infra/recipes/.recipe_deps/recipe_engine/workdir/cache/cipd/infra/tools/luci/cas/git_revision%3Acefd07c708bfd0bb37362a0c90e53fa31b0f8793/.cipd/pkgs/0/_current/cas
Client tool to access CAS.

Usage:  cas [command] [arguments]

Commands:
  help      prints help about a command
  archive   archive dirs/files to CAS
  download  download directory tree from a CAS server.
  whoami    prints an email address associated with currently cached token
  login     performs interactive login flow
  logout    removes cached credentials
  version   prints the executable version


Use "cas help [command]" for more information about a command.

ivan@balha:~/Work/zmeiresearch.github.io/blog$ /mnt/phys2/home/ivan/Work/fuchsia-infra/recipes/.recipe_deps/recipe_engine/workdir/cache/cipd/infra/tools/luci/cas/git_revision%3Acefd07c708bfd0bb37362a0c90e53fa31b0f8793/.cipd/pkgs/0/_current/cas version
0.1

CIPD package name: infra/tools/luci/cas/linux-amd64
CIPD instance ID:  xiOY69QEW4Td0_DKahtTNA30_SiIwg20ei1En_APEUQC

Lovely, just another undocumented tool :) :