Projects are pretty fun. Currently, I restrict myself to small projects (less than 8 hours) because I don’t know if I have the motivation to do more. But I started a small project again today – this morning and it was great. I’ve written quite a few lines of code and they worked pretty fine.

I remember when I first started learning to program. I was happy if a line of code worked after my third try. It was pretty funny. Even a bit later when I first learned C++ I was happy if one or two lines worked without debugging.

One of the most demanding tasks was writing assembler on paper. Actually, this was even included in our finals. The programs were rather simple of course, e.g. a traffic light circuit but thinking about all the registers and stuff was pretty hard. Especially, because I couldn’t test the code.

One thing about writing code is familiarity. And this was one aspect I didn’t consider in the past. When I looked at source code from some experts in a field, e.g. from Peter Norvig I was always overwhelmed. Everything was so clever and well thought through. After I read more and more code from Norvig I’ve learned that he uses the same techniques in different pieces of code. I started to learn his techniques and applied them in my code. And whoosh suddenly it looked also pretty clever.

Also these people are exceptional and have tons of experience. Norvig is in the field over 20 years.

What helped me is learning functional programming languages. Especially my python code improved a ton when I learned Scheme and Haskell. I started to use more list comprehensions and thought more in functional terms. For me, that meant I thought about map, reduce, filter, zip, etc.

A simple example is that you need to manipulate a list of strings for example. The idea is that you want to replace each $ with USD.

A C-type Python-inspired code looks like this:

lst = ["23.56$", "8.43$", "$95.02", "$2.99"]

new_lst = []
for l in lst:
    new_lst.append(l.replace("$", "USD"))

I just didn’t include the very C-ish approach using an index.

A more pythonic approach – which I would probably use – looks like this:

lst = ["23.56$", "8.43$", "$95.02", "$2.99"]

new_lst = [l.replace("$", "USD") for l in lst]

Alternatively, a functional approach:

lst = ["23.56$", "8.43$", "$95.02", "$2.99"]

new_lst = map(lambda l: l.replace("$", "USD"), lst)

I normally, fall back to using map and its friend when I want very performant code because it’s faster.

And I see that a lot of beginners have problems writing pythonic code. It doesn’t help that their teachers often don’t know how to write pythonic code or just write C-like code in Python.

But still it has to do with habit and familiarity. When I’m familiar with a domain I know how to structure my code, how to write functions and how to solve problems. This results in two things: a) I’m pretty fast and b) the code is more robust and nicer.

I think my best-known area of expertise is bots or automation in the web field. Yeah, we say the latter is a better description. I just understand how the web works. How APIs work, how the HTTP protocol works, how JS works, etc. I can quickly do things which other people think is impossible.

It’s funny because I often heard old school hackers talk about younger people who wanted the source code to fix problems while they just disassembled the binary and fixed it there. In the same spirit I see people today who say you can’t write code for some service because they don’t have an API. I just write my own.

When I think back I started writing bots without even using a url library. Everything was sockets and I wrote my HTTP requests by hand. The same for cookie or session handling. Probably, not very people will that do today but I have a great advantage because of this. I have no fear of diving into a new protocol because I know how to deal with sockets. I have no fear of services without an API because I know how to write one myself. Pretty cool.

A few months ago I wondered how BitTorrent deals with resource allocation. I didn’t find so much and I thought about that myself. I think that is just a market. But let’s start at the beginning.

What is BitTorrent?

BitTorrent is a Peer-to-Peer protocol for file sharing. File sharing means down- and uploading files and peer-to-peer means that you don’t connect to a (dedicated) server but to some other person in the network.

However, it neglects one important actor in BitTorrent and that is trackers. If you want to read the BitTorrent specs.

What about trackers?

morehawes.co.uk created an excellent simple overview about BitTorrent and especially trackers. I also took this illustration from there:I quote from the site:

its main role is allow peers to ‘find each other’ and start communication, i.e. to find peers with the data they require. Peers know nothing of each other until a response is received from the tracker. Whenever a peer contacts the tracker, it reports which pieces of a file they have. That way, when another peer queries the tracker, it can provide a random list of peers who are participating in the torrent, and have the required piece.

Which information is available?

Generally we know which pieces (data) a client has and which he needs. In economic terms we have supply and demand.

How is this handled?

There are two main methods used currently. The client requests a random piece or the rarest piece.

Using market mechanism to allocate resources

At the first sight it looks like a market. On the second it doesn’t because you don’t lose your goods if you give it to another (upload one piece). On the third look it looks like a market again because the same applies to digital goods.

So, we have actors in our market and goods (pieces of data). We know that the first actor alone on the market has all data.

What is the objective?

This is a really important question to ask. What do we want? The great thing is that we work with machines (clients) instead of people which means we can build a mechanism which assumes rationality to an extremely high level.

I think the most important objective is distribution. We want a file as quickly distributed and available as possible. So we have two measurements of success:

1. How many actors (client) do have the complete file after some time period?
2. How much data was transferred after some time period?

Why do we need two? The first one helps us to measure the main objective, i.e. file distribution. The second one helps us to get rid of some artificial time period. Let’s say we have an allocation mechanism which allocates 99% of the data as fast as possible but waits until every client has 99%. In this case it would suck on measurement one but rock on measurement two. You can imagine the other way around.

Create prices

Prices are subjective. Clients aren’t. So we have to build some which are subjective to some degree. Most interesting is the subjectivity in relation to needs.

For example, if there are 2000 clients at 95% in the network only you and the original uploader have a piece of data that is missing – all the other missing pieces are already distributed evenly. I think it’s pretty clear that they need this piece from you or the original uploader more than the other pieces.

This is our first factor which I call distribution level.

The other factor becomes evidently if we look at our measurements. We want speed. Again, it’s better that a client which has a high upload speed to get an important piece than one with a low upload speed. I name our second factor speed level.

Problem: No information about speed

Yeah, we don’t know the speed level and even if we knew how fast the internet connect is we don’t know how much a client can send. We have two options at this point:

1. Throw away the speed level and work without it
2. Try to estimate it

Let’s talk about the second option. We know which pieces a client requests, which ones he has, how much he uploaded and downloaded. We can estimate the network speed level. We could try to maximize this with experiments. We could work with assumptions (e.g. avg upload speed ~ 1/8 avg download speed) etc. Because this is more a thought experiment than a real implementation, I will just make speed levels up.

I will stop here and continue this post tomorrow or so.

I’ve written my first line of Python about 8 or 9 years ago. I remember buying a small book about it. Having written some PHP it was awesome. The language looked so clean and intentionally designed.

Then about 6 years ago I dived deeper into Python and used it since then quite regularly. Most of my code was written in Python, then some code in JS, Stata / R and C. Now Python 3 is the official standard and yesterday I read a post called Python 3 is killing Python.

I share one side of the argument which is also represented in this post. That is that Python 3 wasn’t a good thing. There should have been Python 2.8 but I don’t want to talk about it.

This article lead me to revise my belief about languages. I was so deep into Python that I didn’t take a look at new languages like Scala, Clojure, Dart and Go. Heck, I even just wrote my first jQuery code a few weeks ago.

The mystical unicorn

The first times I heard about functional programming must have been in esr’s How to Become a Hacker:

LISP is worth learning for a different reason — the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot.

Norvig’s Teach yourself programming in ten years:

[Learn languages …] one that supports functional abstraction (like Lisp or ML), one that supports syntactic abstraction (like Lisp), one that supports declarative specifications (like Prolog or C++ templates), one that supports coroutines (like Icon or Scheme),

And of course Paul Graham. Interesting enough all are role models for me.

Finally doing it

At the time Haskell became more popular and with it some tutorials. So I started learning Haskell. At the time I’ve written code in PHP, C and Python and the abilities of Haskell were insane. It was so elegant and simple.

Later I wrote some Scheme working through Concrete Abstractions. It was really nice. However, all this code was no production code – just for learning.

So I looked around to see if there are books which are more focused on production and found one on Common Lisp. However, one thing that turned me off was the lack of a standard – not a Lisp standard but a standard work environment. There were thousands of lisp implements and dialects and so I gave it up. Because not writing lisp is easy than writing it.

Going shopping

So yesterday I’ve gone language shopping. I wanted to learn a new one. There were a few names which I’ve seem lately which I wanted to look at closer: Go, Scala, Clojure, Rust and Julia.

My “survey” was totally unscientific and I just looked at the project and wiki pages of each language and decided if I want to know more.

My final two languages were Scala and Clojure. Both run on the JVM which is fast and has tons of libraries. Also Go and Rust are more low-level and Julia is more for scientific calculations.

I looked at code of both and Clojure reminded me immediately of Lisp. It really liked it. Here are a few things which are also great about Clojure – so far:

• Dynamically typed
• Prototyping should be easy
• JVM is fast and tons of libraries
• Code looks great
• REPL
• I never learned about lisp macros

An investment?

I read a few articles on Clojure and one talked about how it’s going to explode. I don’t think so. Lisp exists for such a long time and it never really took off. I think a lot of companies just decide that it’s easier to use C / PHP because there are more hiring opportunities. Also developers still find FP harder. I don’t think that FP languages are inherit hard but that developers start with a C-like language (C++, Java, Ruby, Python, etc.) and the transition isn’t easy.

Would I recommend learning Clojure for a job? Nope. There are basically no jobs with Clojure. To be fair there were also basically none in Python when I first started to learn it. But I think it will stay a niche language. I want to end this with a quote which I already posted but it’s still great:

“So next time I hear the “you can’t get the programmers” line I’m going to respond with something like this:

“If you post an advert for a Haskell developer you will get 20
applicants. All of those people will be the kind of developer who
learns new programming languages to improve their own abilities and
stretch themselves, because nobody yet learns Haskell just to get a job.”

“If you post an advert for a Java developer you will get 200
applicants. Most of them will be the kind of developer who learned
Java because there are lots of Java jobs out there, and as long as
they know enough to hold down a job then they see no reason to learn
anything.”” —haskell.org