back to

2023-01-03 - 2022 devlog

i started a devlog at the beginning of 2022. a year later, here it is:


use this to see which processes are tied to which ports. you need to run it as root to atually see the process id:

i normally use rg. but this is the grep equivalent to `rg -l`:

i also learned that journalctl has a `-f` option for following output. just like `tail -f`:

some memory stuff. first up is free. i feel like i was forgetting about free and using something like htop or top to see the memory. but free is much better.

next up is drop_caches. this is a kernel file. writing to this file triggers a command in the kernel. more memory is usually freed if you run `sync` first:

another kernel thing, but unrelated to memory is strace.


you can convert an epoch time to a normal human readable date with this cli command:


some useful apt logs are sometimes stored here:

simple ubuntu advantage commands:

there are a few aws concepts i want to go over. ami means "amazon machine image". this is a snapshot of multiple ebs volumes and must include a boot volume. ebs stands for "elastic block store" and this would be a snapshot of the volume/hard disk. so if i want to create an image that has all our configuration, i could capture an ebs snapshot of the hard drive.

onto jenkins notes now. here are a few terms to get down:


minikube and kubernetes are pretty much the same thing. minikube is like a beginner friendly version of kubernetes and only runs on your local machine. kubernetes can run on many machines.

with jenkins, the number of executors for an agent shouldn't exceed the number of cpus. also, most things are pretty much located in one directory with jenkins. labels on an agent are useful, because it lets you build on an agent that has "docker" or "maven", or agents can have a label for itself to target a specific build box. a controller and agent should both have the same version of java installed & have java_home setup correctly.

unrelated to that, is an ip address block for private networks. also, for aws security groups, all outbound ports are allowed by default (the app sending requests somewhere else), but all inbound ports are denied by default (no one can access the app).

actually, there are 3 blocks of ip addresses that are reserved for private use. these are:

nacl means "network access control list". like a firewall for things going into and out of a network.


hosted zones in route 53 for aws are just groups for sub domains. for example, "" could be a hosted zone. then records in that zone could be "" or "".


traefik is some sort of reverse proxy thing for orchestration:


show all environment variables for a given process.


jenkins automatically backs up changes to the main configuration in `~/config-history/config/`. useful if jenkins breaks because of configuration issues.


how to run command in jenkins script console:

and a command with a pipe:

how to get details from a certificate:

how to get a java flight recording on a host:


if you want to see the plain text value for all the credentials in jenkins, execute this script in the script console (available to admins):

that hack is taken from here:


is something listening on a port?

in kubernetes, a restart will retain the pods. a redeploy will replace the pods.


how to make puppet read from a different branch:


list files in a tar archive:

jenkins has a reload endpoint that you can hit from your browser if your user has permissions:

here is an initramfs fsck command to fix a filesystem:

one way to restart a bunch of things in docker swarm:

one way to restart one thing in docker swarm:


mongo db crash course:


to send a metric to graphite, use this:

ldap catalog and ldap global catalog are 2 different things. ldap catalog does more stuff and is slower, while global catalog does less and is faster. ldap catalog runs on port 636. ldap global catalog runs on port 3269. so if your ldap connection is slow, maybe switch from port 636 to 3269.


with ansible, you can use this option to start at a specific task:


use this to test if a tcp port is open:

and this to test if a udp port is open:


remove artifactory artifact through api...


some useful commands to check on pbis:

a way to restart pbis:

a way to setup pbis:

a way to add ldap user/group to login access:


to decrypt "enc[pkcs7,...]" things, you need the eyaml program. install it like so:

encrypt and decrypt secrets with:


some rabbitmq terminology:

how to delete all messages in a queue:

in crisis mode, you could block all senders, but allow all consumers. this is how you do that:


in xen orchestration, you can search for a name like so:


to use kubectl in gcp, you can connect using:

after that, kubernetes commands work. here is one simple kubernetes command that gets the namespace:


a few useful consul commands someone showed me:


some useful aws commands:


the "script" and "scriptreplay" commands can be used to record a terminal session. basically the same as asciinema but less hyped:


to enable tomcat debugging, you can add these to the java startup command:

some things that could go in a sudoers file:

and this is a hack to fix brew to work only for your user:


learned more iptables stuff today. some useful commands:


this is pretty good at finding the largest files in a directory:

but this is better, because it gives you folder sizes too:


you can force a bad node in consul to leave the cluster with something like this:

dead nodes will eventually leave, but that makes it actually leave immediately.


when puppet starts, it loads facts first. these are just variables that puppet populates based on files and stuff on the system. you can use the facter command to see the basic values:

for more specific facts that are related to puppet, use the puppet facts subcommand:


lvm is logical volume management. this guide is great:

and this one is good too:

if you have a raw disk, you can create an lvm thing on top of it. or you could create a linux partition first, then install lvm on the linux partition.

commands for setting this up from scratch:

steps for an actual data copy:


here's how you check which package owns a file with apt:


dbus is the thing that replaces the "unix pipe" in your shell. basically just a more standard pipe. systemd depends on dbus and ensures a dbus daemon is executed on startup. dbus is the reason various "ctl" commands work. for example, systemctl commands will send a message to the systemd daemon.

initramfs is a "temporary ram filesystem". when linux boots up, this file system is mounted before the real root filesystem mounts. if the root filesystem has an issue booting, the boot sequence can be halted. busybox is a collection of core programs that the init filesystem might contain. optionally, you could use klibc utils instead of busybox, though klibc utils generally require more memory overall.

this command can show you what the installed dependencies are for a package:

this command can show you what the installed reverse dependencies are for a package:


a few ways to kill jobs in jenkins. you can press the "x" button on the build job. you can go to the build job and append a "/stop", "/term" or "/kill" to the url. if the job still doesn't stop, this command ran in the script console will force the job to stop:


some vault commands for older versions of vault:

you'd need a token for the auth step. this can be found in keeper. don't forget to remove it once you're done!


is docker complaining that there isn't enough space in your little container? no need to worry! this short command will solve all your immediate needs:

start a shell from a docker image:

connect to a shell in a running docker container:


2 great terminal commands i learned about today:


this is cool. you can print out pings that get sent to a box:


useful options on ps:

rss is resident memory, the more useful memory metric.

get all keys from consul, with base64 encoded values:


you can get system properties from a running tomcat instance:

and you can see the currently running java version:

but be careful, because there is an uncommon bug where some jvms will kill themselves when you run those jcmd commands:


how to list enabled services in systemd:

how to list the files that belong to a package for apt/ubuntu:

for newer versions of vault, here are some useful snippets to view secrets:


you can print the contents of a file within a zip archive to stdout with the unzip command:

same command can be used for java jars. for example this will print the java manifest:

you can restore a consul snapshot like so:


with systemctl, you can enable user-level scripts to startup at boot-time & continue after logout with lingering:

arch linux package for mgba currently doesn't support scripting (it's broken for me at least). the app image straight from the mgba website works fine however.


"" in systemd does not mean you have internet connection.

by default in linux, upload/download speed is not limited. but you can limit it with "tc", or use an easier script/wrapper:

this is a pretty good guide on understanding how linux deals with networks:

this ip link command gives cleaner output:

you can see what your router address is with this ip command:

that could also be a good way to check internet connectivity.

arp is the low level protocol that maps ip address to mac address. it can get cached on your computer, so here is a way to delete the cache:

one way to easily spawn many parallel processes and be able to kill them all with ctrl-c is:

though from my trying it, sometimes killing it doesn't work.


bios stands for "basic input output system".

this url has some good details on how drivers work:

run this to get information about a linux driver/module:

a pci device is any device that can connect directly to the computer motherboard via pci slots.

this url has instructions on how to find which device in "dev" corresponds to which driver:

this command will clearly show the module dependencies for a given module:

that command is better than using something like "modinfo", because it shows dependencies recursively and also takes into account aliases.

`lsmod` is basically the same as `cat /proc/modules`, but prettier.

some useful keyboard shortcuts in feh:

the "/sys/bus" directory contains one subdirectory for each of the bus types in the kernel. there are 2 subdirectories in each bus type, "devices" & "drivers".


this is a great paper to help understand sysfs (usually /sys):

this command is really useful:

`lspci` will show the devices connected to the motherboard. the `-k` option shows the current kernel driver loaded for that device as well as the available kernel modules.

this blogpost is also excellent at explaining how to understand your hardware:

`lspci` lists pci busses (plus the device the bus is connected to) and bridges. pci bridges allow you to add more busses. i guess it makes things more modular in a way. if you want to connect to the motherboard, you must use pci. usb was created, because making a cable that uses pci is costly and maybe bulky. a usb cable is much cheaper, though is not as fast as a pci bus. though thunderbolt is comparatively as fast as pci right now. thunderbolt therefore must be built into the computer and can't just be added on.

kernel drivers are code that interacts with hardware. kernel modules are code that can be loaded/unloaded at runtime. most kernel drivers are also modules, but some are statically linked in kernel. for embedded systems, you can statically link all drivers if you want.

`lsmod` or `/proc/modules` shows all the dynamic modules on the system. these is code that can be added or removed during runtime. `/sys/module` shows both all the dynamic modules and modules built into the kernel, so that's why it contains more files than lsmod.


in kernel module programming, there are two special calls you can look for to find where the module code starts and exits:

the kernel also has a function called "request_module":

kernel modules can call that function to load another module. here is an article with more information:

this came up for me because i realized that the "iwlwifi" driver is tied to my network card (lspci -k). but it is also a dependency for iwlmvm. if i unload "iwlmvm", "iwlwifi" also gets unloaded. if i load "iwlwifi", "iwlmvm" also gets loaded. i found out about the "request_module" function by looking at the kernel code for iwlwifi. there is debate about how this functionality can create vulnerabilities, especially for less-maintained drivers.

this is a useful shell snippet to show a graph of all your kernel modules:

you need to display the text in a graphviz visualizer like:

in firefox, use the "*" prefix to search bookmarks and "^" to search history.

in kernel module development, instead of using `module_init` and `module_exit`, you could also just use `module_driver`. that's a macro that just does the very basic work for `module_init` and `module_exit`.

here's a short book that goes over customizing your own kernel:

here is a quote about kernel modules:

taken from here:

so technically, you could get a slightly faster/memory efficient system if you don't use kernel modules. that's assuming your hardware doesn't change too.

configuring the kernel is similar to buildroot (it seems like buildroot is just a layer on top of the kernel configuration honestly...). if you have the kernel source code on your machine, you can run:

and that gives you the nice gui with all the kernel options inside you can select from. it saves your configuration in a `.config` file. the kernel usually saves the configuration it was built with in the /proc/config.gz file. so if you build the kernel with the contents of that gzip file, you should have an identical kernel to the one you currently use.

the `/lib/modules/<kernel-version>` directory contains all the kernel modules included with your kernel. depmod will get a list of all those modules and put them in the `modules.dep` (human readable) and `modules.dep.bin` (modprobe readable) files. modprobe can use those files to see figure out dependencies when loading new modules. is the linux weekly news. the articles are pretty good there too.

there is an "init_module" (not to be confused with module_init) system call. that system call is what actually loads kernel modules. so a tool like modprobe probably calls that systemcall (or finit_module) under the scenes).


this command will give you the vendor & device id for your pci devices:

to get usb vendor & device ids, just use lsusb.

when you have the vendor & device ids, you can grep through the kernel source code with the vendor id first, then the device id. by going through the code, you can hopefully find which driver is associated with that device.

if kernel code is a module, you can set module parameters with modprobe. eg:

if the kernel code is built into the kernel, you'd need to pass the parameter into the kernel before it loads (bootloader). the parameter would look like "<module-name>.<option>=<value>". eg:

udev stands for "userspace /dev". it used to be an independent project, but eventually merged with systemd. here's a funny post about udev:

/dev/shm is similar to the /tmp directory. they are both world writable. the difference is that /dev/shm is *always* stored in ram (if available to your distribution) and the available space is configurable. "/tmp" on the otherhand is usually stored in ram, but is configurable by the user to store it on disk too. better to rely on /tmp, because that is more likely to be available.

systemd has a concept of "api filesystems". these are filesystems that systemd mounts early on to specific places, without needing to specify them in fstab. though you can override the configuration for many of these filesystems with fstab. more description here:

the `mount` command will show you all the mounts, including those "api filesystem" mounts. by default, df doesn't show you all the mounts. but you can pass the `-a` option to show all the mounts as well. there is also the `-t` option, which will show the file system type as well:

here are the different filesystem types mounted for my laptop as an example (coming from the `-t` option):

if you want to know about one of the file systems, the best place might be the kernel source, under the `fs` directory. specifically, check the `kconfig` files.

`binfmt_misc` is a cool kernel module. it makes it so you can execute files that require an interpreter, without needing to specify the interpreter, like python, java, or exe files. to use it, you need to make sure the module is loaded, make sure it's mounted to /proc/sys/fs/binfmt_misc.

systemd automounts use the kernel's "autofs". when enabled, automounts will display an "autofs" mount with the mount command or dashes with df:

if you try to actually access that directory, then the autofs kicks in and actually mounts it right as you try accessing it.

you can create character and block devices with the `mknod` command. this is mostly replaced by udev, because udev automatically creates character and block devices in the /dev directory for most linux distributions. anyways, to create the character or block device, you pass a file name, type (character or block), device major version (mapped to a driver), device minor version (used by the driver).

to help with writing udev rules, you can use this command:

it will give you the values for different udev attributes.


`fbv` is a linux framebuffer image viewer. that's cool.

you can run an sdl application in the linux framebuffer with the environment variable `sdl_videodriver=kmsdrm`. input requires libevdev and for your user to have read access to `/dev/input/*`.

`evtest` is a cli utility part of evdev. you can use it to test things like a mouse and keyboard. just run `evtest` without parameters to see available inputs to test on.