Jakob Stoeck

Download and query all domains of the world-wide web

2023-01-15T01:31:00+00:00

I was wondering whether I could get a list of all world wide web domains on my local laptop and query them, like a phone book. As it turns out, that is straight-forward: You can download all top-level domains (TLD) zone server zonefiles, then use local tools to search through the less than 10 GB gzipped files.

How to download

Since domains are stored online in domain name system (DNS) servers that are reachable by the public, you could use the DNS protocol to request for the full list of domains of each DNS server that is responsible for a top-level domain (like .com). But for security reasons, you cannot request a zonefile from an authoritative DNS server anymore as this would enable an amplification denial-of-service attack.

That is why the Internet Corporation for Assigned Names and Numbers (ICANN) provided a Centralized Zone Data Service, where any interested party can request and subsequently download nearly all zone files. For the vast majority of the 1,000+ TLD you don’t need a better reason than “for research purposes”, whereas some of the more obscure ones request for more information.

To download all zonefiles for the as of writing of this post 266,372,902 domains (easy to remember: slightly less domains than c, the speed of light):

Go to https://czds.icann.org/ and register
Go to https://github.com/icann/czds-api-client-python, input your credentials as mentioned in the readme, install dependencies and download:

$ cd czds-api-client-python
$ cp config{.sample,}.json
# update config.json with your credentials
$ pipenv install requests
$ pipenv run python download.py

How to query

Now, you will have more than 1,000 gzipped text files with all domains. The files are tab-separated. DNS has entry types that mark different purposes: mail servers, name servers (what your browser uses), signatures, etc.

For example, the first 5 lines of the .pizza TLD look like this:

$ zless pizza.txt.gz | head -n5
pizza.	3600	in	soa	v0n0.nic.pizza. hostmaster.donuts.email. 1672103559 7200 900 1209600 3600
0.pizza.	3600	in	ds	27676 8 2 95C9CB61DEF5016F2905C7478044E011AE3C041AEB31E5BC80C11DEE7FC05142
0.pizza.	3600	in	ns	ns-cloud-d1.googledomains.com.
0.pizza.	3600	in	ns	ns-cloud-d2.googledomains.com.
0.pizza.	3600	in	ns	ns-cloud-d3.googledomains.com.

For the reason of this project I’m interested in the name servers which are marked with ns in the fourth column. You see that for the first lines of pizza.txt.gz that is the domain http://0.pizza/ (Which is defunct. What did the owner want to do with that domain? Do they sell pizzas in the shape of a zero with a hole in the middle? We will never know.).

To get all domains from a name server (the ones your web browser would use), we filter on the fourth column, remove duplicates, and send them to fzf (on mac: $ brew install fzf) for fuzzy searching capabilities:

gzcat *.txt.gz | awk -F '\t' '$4=="ns" { print $1 }' | uniq | fzf

Turns out this is fast enough to checkout the data.

Searching through all domains worldwide instantly is pretty fun! There are so many fun, surprising and head-scratching domain names out there. I hope you find some which excite you!

Some observations

People use DNS for much more than I thought.

Especially public signatures, see an example from .pizza:

$ gzcat pizza.txt.gz | awk -F '\t' '{ print $4 }' | sort | uniq 
a
aaaa
dnskey
ds
ns
nsec3
nsec3param
rrsig
soa

There are so many TLDs!

A bit less than 2,000 TLDs, with even local variants like .vermögensberater (financial advisor). You can lookup the history and raison d’être of each TLD on https://icannwiki.org/.

The first domain was `.no`

Limitations

The zone files do not include subdomains. Some TLDs use third level domain names, like .co.no. These subdomains are not on the TLD zone server either, and hence not part of this list. Non-ascii domains are in puny code, and not translated. You would need to convert either your search or the index with something like https://pypi.org/project/idna/ or https://docs.rs/idna/latest/idna/.

Stuff I used in January 2019

2019-01-02T21:08:00+00:00

Quick recap of noteworthy tech stuff I used:

1. Pipe `tcpdump` from remote server to local Wireshark

This will show all the traffic from remote on your local Wireshark application. brew install wireshark before if you haven’t done, yet:

ssh username@remote 'tcpdump -s0 -nn -w - not port 22' | wireshark -k -i -

See the not port 22 which makes sure to not track your ssh traffic your sending from the remote to the local server.

2. CSP: Content Security Policy

A HTTP Header to whitelist script locations so you mitigate XSS attacks. See the MDN Website for details., e.g. if you want all your content come from your domain, excluding subdomains, you should send this HTTP header: Content-Security-Policy: default-src 'self'.

3. Format JavaScript Objects as JSON with jq

jq can only format JSON but not plain JavaScript objects. To convert the latter to the former I used “relaxed JSON” (rjson) like this:

brew install rjson jq
rjson <(./returnsJavaScriptCmd) | jq

4. Draw diagrams with yEd

It seems that yEd is the best editor for technical diagrams out there. If not dot files go a long way and there is an easy “GraphViz for the poor”, i.e. viewing dot files without the graphviz app. It is using the excellent entr to run automatically on file change:

brew install entr dot
echo myfile.dot | entr -c dot -Tpdf -o /tmp/dot_out.pdf /_

5. Flutter

Write native Android apps in Dart.

6. PostgREST

Serve a RESTful API from any database with the great PostgREST. Death to the ORM and all its business code which is duplicating database structures. Here you use all the power of a great database to build up a CRUD API including authorization. Declarative programming ftw!

7. GreenSock Animation Platform

GSAP is professional-grade animation for JavaScript canvas and looks amazing.

8. Jaeger

Tracing in distributed systems with jaeger.

9. Update your private key

If your private key is still in an old or unsecure format, you can update it with:

ssh-keygen -p -o -f PRIVATE_KEY

10. Business Process Modelling Language

Model your business process in BPML instead of writing business logic in code. Old-school tools like Activiti or jPBM support them and there are also fancy new ~~unstable~~ nodejs packages for it. This makes it easy for your stakeholders to define and change business processes without you having to change that in your code.

Don’t demolish

2018-02-18T13:00:00+00:00

Transformation de 530 logements, bâtiments G, H, I, quartier du Grand Parc - Lacaton & Vassal, Druot, Hutin, Image © Philippe Ruault - For all uses, thank you to contact him

Much is written about how you should not rewrite a whole application and I have added some further reading at the bottom of this post. Here I want to add some reasons of why you probably landed where you are right now and why a rewrite is just another indicator of your fundamental problems:

TL;DR: Software development teams should solve business problems instead of implementing predetermined solutions, need performance indicators and architectural principles.

An analogy in civil engineering

Housing complexes built in Europe during the 1960s and 1970s are largely considered outdated and urbanistically failed. France ruled in 1990 to rebuild them. Until now, the demolition of 113,200 housing units and the relocation of its inhabitants cost 3 billion. Building 105,000 new homes cost 12 billion - a total of 15 billion for a loss of 8,200 homes.

A couple of architects were offended for economic and social reasons and started to show the government how to renovate instead of demolishing and letting the residents live in their home. There is a great exhibition about it.

In one example at Cité du Grand Parc in Bordeaux, three old housing complexes with 4,000 inhabitants were augmented by these architects by adding winter gardens to the front and back extending the living space by one third. Serving as a climate buffer they additionally reduced the house energy consumption by half and added a better working ventilation. The cost came down to half of what a complete rebuild was estimated and renovation was organized in a way to let the residents live inside their homes nearly the whole time.

The blue winter gardens were added to the structure for more living space and climate buffer, making a demolition and relocation unnecessary, Image © Lacaton & Vassal - Druot

Back to software engineering

So you have a “legacy application” which needs a rewrite. Instead of throwing it away and commencing the rewrite ponder about how it could come so far and whether your current organization is really capable of writing something better.

If your current application is beyond repair in your eyes, the fault most probably lies not within the technology but your organization. Instead of rewriting, think about the prerequisites which are not fulfilled.

1. Product development done wrong

Many software organizations use agile methods nowadays. Agile teams should focus on business problems. If you don’t let them, they cannot iterate and you just have a badly implemented waterfall team in disguise.

As an example: You have an online shop and too many abandonments before checkout.

Do: Let the team solve business problems

The agile approach is to have an interdisciplinary team e.g. “Customer Journey” and hand them this problem without a solution. Only now is this team in the position to iterate on ever-enhanced solutions: tighten the funnel, make user studies to search for unknown problems, install monitoring, data analysis etc.

Henrik Kniberg (2016) Source

The team remains active also if there are no current problems which are handed to “Customer Journey”. They just enhance the current state of things: Better monitoring, faster performance, more tests, more data analysis, fix small problems before they become big, etc. Top management sometimes focuses on utilization and thinks this approach is inefficient. This is a fallacy. You don’t improve the flow of traffic on a highway by improving its utilization (i.e. introducing more vehicles).

Don’t: Let the team implement a given solution

This is different to a lot of organizations where the business and technology leaders come up with a solution beforehand (“The funnel must be shorter”) and let a team implement this. Many organizations using Scrum do this: They are responding to feedback instead of solving problems.

This is not agile even if you have agile teams. See how the team cannot iterate on the solutions anymore but just on a single predefined solution which might even not accepted until the last iteration. This is a waterfall-y process just much less efficient.

Henrik Kniberg (2016) Source

If you are doing this you should not rewrite your application. It will end up the same like the last one. Instead rewrite small parts of it. And try to change the organizational structure to get more into the best practices outlined above.

2. Performance indicators to improve are not clear

You must have a strict definition and monitoring of performance indicators. If you don’t then the next application version will have similar business problems like the current one.

Do: Applications report their status by themselves

All components should have business monitoring capabilities built-in. Too many errors, checkout abandonments or fluctuations during the current time period? The application itself should report this to a monitoring solution.

Like this you have much more context information to what is happening. You need this to always be able to enhance parts of the application where needed.

Your data analysis is much more profound if you have this internal monitoring and changes are seen directly. You allow teams to be creative to improve the given KPIs bit by bit.

Don’t: Only do external monitoring

If you don’t have extensive monitoring capabilities built into your application which are adapted to your business case but are only monitoring from the outside with some analytics tool you do not have enough insights to rewrite your application.

It will end with similar shortcomings like the current one. Instead focus on setting clear business goals, carve out parts of your software and organization which optimize this KPI and steadily improve them.

3. Architecture Principles

For your problem solving teams to work autonomously and efficiently, your organization needs to commit to architectural principles which are measurable and testable (around ten is a good size). This cuts down on wasted communication around implementation details. Depending on your business these may be different.

Do: Make teams adhere to architecture principles

The art of scalability has a great starting list. Adjust the list to fit your organization. More details can be found in the book (chapter 12), but this should give you a good idea:

Build small, release fast, fail small
Design to be monitored
Design to be disabled
Design to rollback
Use mature technologies
Buy when non-core
Commodity hardware
Isolate faults
N+1 design
Asynchronous design
Automation over people
Horizontal scaling
Design for multiple live sites
Stateless systems

Check new solution concepts from your teams against your architecture principles in a small informal architecture board and give suggestions to fulfill them when needed.

Don’t: Let a team overlook architecture principles

If each team decides themselves which principles are important you are creating a lot of communication deciding about who and how something should be built instead of what should be built. While the former is inefficient, error-prone and may lead to intra-team fights the latter may be very productive.

Instead build up architecture principles together with your teams to which each one adheres to.

Conclusion

Teams should solve business problems, need performance indicators and architectural principles. If you don’t have them don’t even think about a rewrite.

If you do, read on why you still should not rewrite and instead shape the organization and contingent software to improve over time:

Before and after augmenting a building using its main structure, Image © Philippe Ruault - For all uses, thank you to contact him

(Edit 2018-03-07: Added civil engineering analogy and changed title from “rewrite” to “demolish” to make the intent clearer.)

Fireside Chat With Joel Spolsky

2018-01-23T19:43:00+00:00

Just came back from the fireside chat with Joel Spolsky organized by Stylight’s Johann Romefort in the co-working space Mindspace in the vicinity of Munich’s Viktualienmarkt. For those of you who could not attend but are interested these were some of the topics:

Mindspace co-working with fireside chat (Jakob Stoeck)

Evangelists become more important

An ever-increasing amount of smaller companies employ evangelists to push products or services like their API or technology.

Glitch - Learn to code

A product from Fog Creek which debuted in 2016: Glitch. A “friendly community where you’ll build the app of your dreams”. Looks fun and is educating beginners about “real programming” (Joel) unlike “dumbed down” children’s apps which just teach you basic concepts. It teaches developing in JavaScript on Node.js with transparent deployment and provisioning so that beginners don’t have to learn how to use git, GitHub, deploy workflows etc. before starting.

Company-internal Stack Overflow

Stack Overflow will get a company-internal version which might be on-premise or cloud-hosted. For companies starting around 500 employees you can have your own private Stack Overflow and exchanging questions and answers. A very interesting twist to the ever-outdated wiki or other knowledge solutions.

Python - fastest growing language

Asked which technology trend might be important in the coming years Joel stated that according to Stack Overflow statistics, Python is the fastest growing language because of many Big Data and ML applications. Nothing new here :)

StackExchange and Stack Overflow

The importance of the 2018 developer survey in the industry is reiterated. Participation especially in Germany is rare. Joel is asking for help and more people taking the survey.

The answer for the question which community is the most fascinating on StackExchange Joel replied with Math Overflow with discussions ranging qualitatively between a math book and a journal. Seems like most important mathematicians are online and discussing on there. Go on, how many of the top questions do you understand?

The first user ever in a StackExchange community cracked a reputation of 1 million. Who is it? Yes, you’re right. Living outside of London he is using his commute time five times a week to help the community. Nice! Here, have some Jon Skeet Facts.

Outsourcing and remote work

Like many people in the industry Joel is no big fan of outsourcing developers. It really just shifts the burden of finding great developer talent to finding a great recruiter who finds great developer talent which might be even harder. In both cases you need to interview the candidates yourself so there isn’t even a time saving.

That doesn’t mean that you shouldn’t work with people from remote. On the contrary, you should to find exceptional talent. But your whole communication needs to switch to remote, no mix-and-match. If you hold a meeting, it is a remote meeting for everyone. Even if some people are sitting in the same office, they go to their phone booths or private rooms to use the same technology like everyone to make sure that the quality is always high and you don’t have second class citizens communicative-wise. Alternatively pay big sums, but there were some laughs to the last quote:

“If you have some walls then you don’t need to pay so much!”

Facebook’s headquarters in Menlo Park (Washington Post)

It was a nice little after work meetup with many interesting guests. Thanks again!

Open Source Speech To Text App

2017-09-14T07:30:00+00:00

Playing a bit with Java and Swift, I built two small apps for iOS and Android and solved a problem of mine: With “That’s What She Said” you receive the text of any voice message without listening to it.

Source Code on GitHub: https://github.com/jakobstoeck/

How it works

You need to send the voice message you have to an iOS action. Depending on the app this opens behind a button called “Share”, “Forward” or “Export”.

Often it’s also this share symbol.

If you tap on it, the iOS actions open where you can choose That’s What She Said:

The transcribed text is shown after a few seconds as a notification:

Supported Apps

Most apps should word since it only needs to support iOS actions. Those appear usually if you tap on “Share”, “Forward” or “Export”. I tried it succesfully with:

WhatsApp
Telegram
Voice Memos

One notable exception is the Apple Messages app. I couldn’t find any way to share or export a voice message in this app.

Privacy

Depending on the audio format voice is either sent directly to Google’s (for Opus and Flac) Speech Recognition service or Apple’s (all other audio formats). No other data is transmitted or stored.

RAM disk for faster applications under macOS

2017-06-13T15:13:00+00:00

Theory: A RAM disk might speed up IO-bound computations under macOS.

Experiment: Find maximum speedup

Theoretical maximum. I use pv ($ brew install pv) to measure the throughput of writing data and then use the following options to get the average data througput for 450 MBs written: pv --average-rate --stop-at-size --size450m or in short pv -aSs450m.

Contrast data throughput to the following sinks:

the void
SSD
RAM disk
RAM disk mounted with OS mount command
RAM disk folders mounted with bindfs

the void

This should be the fastest possible data rate: Piping zeroes into the void. For bigger files than 450 MB the troughput would be even higher, maxing out at around 17GiB/s.

$ pv -aSs450m < /dev/zero > /dev/null
[13.5GiB/s]

SSD

My MacBook Pro Late 2012 SSD has a little over 300MiB/s write throughput in this test.

$ for i in {1..10}; do pv -aSs450m < /dev/zero > test; rm -f test; done
[ 307MiB/s]
[ 300MiB/s]
[ 305MiB/s]
[ 305MiB/s]
[ 303MiB/s]
[ 299MiB/s]
[ 309MiB/s]
[ 303MiB/s]
[ 306MiB/s]
[ 304MiB/s]

RAM disk

First setup the ram disk:

$ VOLUME_NAME=RamDisk
$ SIZE_IN_MB=1024
$ DEVICE=`hdiutil attach -nobrowse -nomount ram://$(($SIZE_IN_MB*2048))`
$ diskutil erasevolume HFS+ $VOLUME_NAME $DEVICE

The DDR3 RAM with 1600MHz in this setup has a throughput of ~1.38GiB/s. Nearly 5 times the SSD.

$ cd /Volumes/RamDisk
$ for i in {1..10}; do pv -aSs450m < /dev/zero > test; rm -f test; done
[ 1.3GiB/s]
[1.29GiB/s]
[1.37GiB/s]
[ 1.4GiB/s]
[1.35GiB/s]
[1.36GiB/s]
[1.39GiB/s]
[1.39GiB/s]
[1.38GiB/s]
[1.37GiB/s]

RAM disk mounted with OS mount command

Applications write their cache inside files and folders. One way to move those folders to the RAM disk would be to replace them with symbolic links to the new destination on the RAM disk. This has the big disadvantage that the folders would stop working when the RAM disk is down (e.g. during/after a restart or when ejecting the ram disk).

A more robust solution is to bind the folder to a folder on the RAM disk. Under linux the mount --bind command can do this, under macOS this is not possible with single folders but only with whole devices: You can bind a folder to a device, i.e. the ram disk, but not to another folder on the ram disk.

A trick is to create the base folder structure of the future ram disk in a folder on the harddisk and mount the ram disk over this folder. This way, even when the mount point is down, folder access gracefully falls back to the hard disk folder structure. Which is very important for some folders like /tmp which are needed during startup.

$DEVICE=/dev/disk3 # return of hdiutil
$MOUNT_POINT="$HOME/mounts"

$DIRECTORIES='/private/tmp'
for f in $DIRECTORIES; do
	mv -v $f $MOUNT_POINT$f
	ln -vs $MOUNT_POINT$f $f
done

newfs_hfs -v `basename "$MOUNT_POINT"` $DEVICE
mount -o noatime -t hfs $DEVICE $MOUNT_POINT

for f in $DIRECTORIES; do
	mkdir -vp $MOUNT_POINT$f
done

cd $mount_point
for i in {1..10}; do pv -aSs450m < /dev/zero > test; rm -f test; done
[1.29GiB/s]
[1.43GiB/s]
[1.43GiB/s]
[1.43GiB/s]
[1.34GiB/s]
[1.37GiB/s]
[1.47GiB/s]
[1.46GiB/s]
[1.42GiB/s]
[1.36GiB/s]

RAM disk folders mounted with bindfs

In the last chapter we looked at how Linux can mount --bind single folders. Under macOS we need to install bindfs to be able to emulate it:

$ brew cask install osxfuse && brew install bindfs
$ mkdir ~/testdir
$ bindfs /Volumes/RamDisk/testdir ~/testdir

With our usual test we see that, unfortunately bindfs is much slower than even using the hard disk:

$ cd ~/testdir
$ for i in {1..10}; do pv -aSs450m < /dev/zero > test; rm -f test; done
[ 143MiB/s]
[ 149MiB/s]
[ 148MiB/s]
[ 147MiB/s]
[ 145MiB/s]
[ 138MiB/s]
[ 145MiB/s]
[ 147MiB/s]
[ 146MiB/s]
[ 146MiB/s]

To unmount the bindfs volumes it’s easiest to go to $ open /Volumes/ and click on the eject symbol.

Conclusion

Binding folders to a RAM disk significantly speeds up the write throughput of the bound folders. You may use it for hot paths in your file system where many files are constantly read and written.

Which folders may be sped up?

$ sudo fs_usage -w -t 5 -f filesys | tee fs_usage.log | egrep -o '(/.+?) {3}' | sed -e 's/\/dev\/disk[^ ]+  //' | sort | uniq -c | sort -nr

Websites which work great without JavaScript

2017-06-01T11:35:00+00:00

Contrary to popular belief, the web still works quite well without JavaScript:

These work with some functionality missing

bing.com (no images, no login)
espn.com (some images missing)
GitHub.com (read-only browsing works, opening PRs and the like not)
GitLab.com (read-only browsing works, drop down menus are not)
meetup.com (no RSVP)
tumblr.com (blogs work, the search does not)
WashingtonPost.com (some images missing)

Do not work

any video except gifs (YouTube.com, Twitch.tv, even native HTML5 videos need JavaScript for the play event. There are applications which support YouTube URLs though, like VLC: vlc 'https:// www.youtube.com/watch?v=somevideoid', though)
instagram.com
linkedin.com
maps.google.com
Netflix.com
Reddit.com (mobile version. alternative: use the desktop version)
taobao.com
ted.com (video is not playing, download button does not work either, but that could be fixed easily)
translate.google.com (error 403)

Why disable JavaScript?

Faster browsing

Loading times are much lower since JavaScript files are not requested. Rendering times are also faster because no scripts are executed during this phase. Most pages are finished in under one second.

Less CPU usage

Besides the initial loading, many sites are continuously executing scripts which put load on the CPU. Badly written pages even more so.

More battery on mobile

Mobile surfing holds up much longer.

Less intrusive animations

No “subscribe to our newsletter” popups, no scrolling-based “this might interest you” notifications. No changing of scrolling behavior.

How to disable JavaScript

Go cold turkey:

Safari: Develop > Disable JavaScript
iOS: Settings > Safari > Advanced > JavaScript
Chrome: Preferences > Show Advanced Settings … > Content Settings … > Do not allow any sites to run JavaScript
Firefox: about:config > javascript.enabled = false

Pro Tips:

For Safari under macOS you can set a Keyboard Shortcut to toggle JavaScript under System Preferences, e.g. Cmd-Shift-J
For Chrome you can add exceptions to sites you like to always run JavaScript

Alternatively use an extension like NoScript.

If you want to add a site or change something just edit this page. With JavaScript since GitHubs commit button does not work without it. If anyone from GitHub is reading this, it would be great if you could fix this 😉

How to encrypt RTMP (or anything really) over SSL/TLS

2017-05-16T13:36:15+00:00

You want to communicate over a secure channel but your client application doesn’t support it? That’s a job for a transparent SSL proxy.

This articles describes on how to use stunnel as a transparent proxy to encrypt communication. With a transparent proxy you do not need to change the client application.

Problem

You want to send data securely between two computers over the internet but your sender application does not support SSL.

Example

You want to stream a video with ffmpeg over RTMP to an RTMP server like Wowza or nginx with an rtmp module. While ffmpeg supports RTMP it does not support the SSL-encrypted version RTMPS.

We use stunnel to wrap the RTMP stream with SSL/TLS (“SSL proxy”) and deliver it to the RTMP server. The server uses stunnel again to decrypt the RTMP stream.

Now the sender acts as if it supports encryption and decryption. In the case of ffmpeg it would send an encrypted stream when the SSL proxy is active and an unencrypted one when the SSL proxy is inactive:

$ ffmpeg -re -i "~/some.mp3" -acodec copy -f flv 'rtmp://www.example.org:443'

Solution

The needed steps are:

Install stunnel on a server called gateway
Route the sender traffic to the gateway without changing the receiver destination
Configure stunnel to re-route the traffic to the receiver address after SSL-wrapping it

1. Install stunnel on a server

Stunnel runs on all popular operating systems. To install it use your package manager, e.g. on macOS brew install stunnel. Stunnel accepts an incoming connection on a specified port, encrypts it and send it to another specified address.

The default configuration location is /etc/stunnel/stunnel.conf (or $(brew --prefix)/etc/stunnel/stunnel.conf on macOS). Change its content to

# /etc/stunnel/stunnel.conf
foreground = yes
debug = 5

[my-encryption-service]
client = yes
accept = 1234
connect = www.example.org:443

Stunnel now listens to traffic on port 1234, encrypts this traffic and routes it to www.example.org on port 443. For further explanation of the config parameters check $ man stunnel. You can test the configuration by starting stunnel and on the same machine connecting to port 1234 (in another terminal window since stunnel is set to foreground = yes):

$ sudo stunnel
$ echo "hi" | nc 127.0.0.1 1234

The stunnel log should then show that you successfully connected to the remote server over a secure connection:

[…]
LOG5[0]: Service [my-encryption-service] accepted connection from 127.0.0.1:53870
LOG5[0]: s_connect: connected 93.184.216.34:443
LOG5[0]: Service [my-encryption-service] connected remote server from 192.168.0.1:53871
LOG5[0]: Connection closed: 3 byte(s) sent to TLS, 0 byte(s) sent to socket

You could stop here if the destination addresses were known beforehand and could be written in the config. In our example that’s not the case so proceed to step 2:

2. Route the sender traffic to the gateway without changing the receiver destination

Now that stunnel works, all the to-be-encrypted traffic needs to be directed to the stunnel port transparently, i.e. without the sender knowing that its traffic gets redirected. This can be done by setting the stunnel server IP as a gateway for the client.

On the sender: Setting a gateway or router IP is usually done in the network configuration. Depending on the client application or operating system that is done in different ways. Search for “set default gateway” and add the operating system your sender runs on if you’re not sure. Then set the “gateway” or “router” IP to the stunnel server from step 1. Now all the traffic which is sent from the client is routed to the gateway address.

On the gateway: The gateway needs to be configured to not drop those packets but to route them to their destination. This is called “IP Forwarding” and should be enabled. Under Linux that can be done with $ echo 1 > /proc/sys/net/ipv4/ip_forward. Search online for other operating systems.

Now the client computer should still be able to normally access the network but all its traffic is routed over the gateway before accessing the internet. Which enables us to do the last step:

3. Configure stunnel to re-route the traffic to its original destination after SSL-wrapping it

Instead of hard-coding a destination like we did in step 1, change the stunnel configuration to read like this:

# /etc/stunnel/stunnel.conf
foreground = yes
debug = 5

[my-encryption-service]
client = yes
accept = 443
transparent = destination

The accept port should be equal to the port on the receiver side and we do not have a hard-coded destination anymore but instead use the original destination.

On the gateway: Route the incoming traffic which is directed to port 443 to the local port 443 of stunnel. To do this you need to know how your network is named (look at $ ifconfig, in this example I use “eth0” and a Linux gateway):

$ iptables -I INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
$ iptables -t nat -I PREROUTING -p tcp --dport 443 -i eth0 -j DNAT --to-destination [youriphere]:443

Make it secure

Now you should be able to connect to any service with your sender over an encrypted connection as long as you use the configured port (443 in our example) in the destination address.

To prevent man in the middle attacks, tell stunnel which certificates to accept by concatenating all the valid certificates in a file and add this to the config:

# /etc/stunnel/stunnel.conf
verify = 4
CAfile = /etc/stunnel/your_certificates.crt

Ask the receiver maintainer which certificates are valid or have a look yourself at the current certificate chain with this command:

$ openssl s_client -showcerts -connect www.example.org:443 </dev/null

Limitations

If you need SNI this approach might not work with all protocols without additional work. The reason is that for SNI you need the destination domain which is written inside the application protocol and must be read by stunnel. But stunnel does not understand all protocols (it normally doesn’t need to for just encrypting it). It does understand some protocols, though, like imap, http, proxy, smtp and others.

The performance of stunnel is nice, but you need to pay attention to some settings like ulimit and which ciphers you use.

Updates for macOS

2017-04-25T12:00:01+00:00

Installing and updating system packages and applications is much easier under macOS using the CLI if you use homebrew and the Mac App Store Command Line Interface, then you can just:

update all the system components,

$ softwareupdate -i

update all the apps installed via the App Store and

$ mas upgrade

update everything installed via brew and cleans up the otherwise always growing cache folder

$ brew upgrade --cleanup

Now you have a completely up-to-date machine. I added this with some additional upgrade commands to my .zshrc to have a shell function for it, so I can just type $ upgrade:

upgrade() {
	# updates system updates, mac app store, homebrew packages and ruby gems
	softwareupdate -i
	mas upgrade
	brew upgrade
	brew cleanup -s
	for app in `brew cask outdated --quiet`; do brew cask install --force $app; done
	brew cask cleanup
	gem update --no-document
	gem cleanup
}
outdated() {
	softwareupdate -l
	mas outdated
	brew outdated
	brew cask outdated
	gem outdated
}

Jakob Stoeck

Download and query all domains of the world-wide web

How to download

How to query

Some observations

People use DNS for much more than I thought.

There are so many TLDs!

The first domain was .no

Limitations

Stuff I used in January 2019

1. Pipe tcpdump from remote server to local Wireshark

2. CSP: Content Security Policy

3. Format JavaScript Objects as JSON with jq

4. Draw diagrams with yEd

5. Flutter

6. PostgREST

7. GreenSock Animation Platform

8. Jaeger

9. Update your private key

10. Business Process Modelling Language

Don’t demolish

An analogy in civil engineering

Back to software engineering

1. Product development done wrong

Do: Let the team solve business problems

Don’t: Let the team implement a given solution

2. Performance indicators to improve are not clear

Do: Applications report their status by themselves

Don’t: Only do external monitoring

3. Architecture Principles

Do: Make teams adhere to architecture principles

Don’t: Let a team overlook architecture principles

Conclusion

Fireside Chat With Joel Spolsky

Evangelists become more important

Glitch - Learn to code

Company-internal Stack Overflow

Python - fastest growing language

StackExchange and Stack Overflow

Outsourcing and remote work

Open Source Speech To Text App

How it works

Supported Apps

Privacy

RAM disk for faster applications under macOS

Contrast data throughput to the following sinks:

the void

SSD

RAM disk

RAM disk mounted with OS mount command

RAM disk folders mounted with bindfs

Conclusion

Which folders may be sped up?

Websites which work great without JavaScript

These sites work great with all essential functionality

These work with some functionality missing

Do not work

Why disable JavaScript?

Faster browsing

Less CPU usage

More battery on mobile

Less intrusive animations

How to disable JavaScript

How to encrypt RTMP (or anything really) over SSL/TLS

Problem

Example

Solution

1. Install stunnel on a server

2. Route the sender traffic to the gateway without changing the receiver destination

3. Configure stunnel to re-route the traffic to its original destination after SSL-wrapping it

Make it secure

Limitations

Updates for macOS

The first domain was `.no`

1. Pipe `tcpdump` from remote server to local Wireshark