Using SQLite by Jay A. Kreibich

I’ve used SQLite in multiple projects and loved it. It’s fast enough for my applications: Appropriate Uses for SQLite.

Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite. The 100K hits/day figure is a conservative estimate, not a hard upper bound. SQLite has been demonstrated to work with 10 times that amount of traffic.

I first started using SQLite with Python because I thought there must be a better alternative to writing/reading a file for storage. That was SQLite which comes included with Python. Also I use SQLite basically for every web project I start – just because normally the project doesn’t scale over 50-100k hits, so I don’t really care about my DB as long as it works and is easy to set up.

I noticed in the last few months that I never really dove into databases. I want to change that and finally read a book from cover to cover about a database. I decided to choose SQLite just because I use it mainly. However, I still have one book in my reading list which goes into more depth regarding database design.

In-memory databases are often used to cache results pulled from a more traditional RDBMS server. An application may pull a subset of data from the remote database, place it into a temporary database, and then process multiple detailed searches and refinements against the local copy. This is particularly useful when processing type- ahead suggestions, or any other interactive element that requires very quick response times.

I never thought about doing this but I think it’s a nifty idea. You can easily create in-memory databases with SQLite and the interface can remain the same.

SQLite offers automatic indexing if you define a explicit primary key. You can still define other indexes for yourself.

I also learned that you can do the following:

INSERT INTO table_name (col1, ...) SELECT ...

Which allows you to copy a database or create a new one on the fly.

Also I learned a bit more about transactions. I only read about it especially in regarding to Clojure and STM funnily enough. But it was interesting to see that you can use savepoints, releases and different locking mechanism with SQLite.

There’s one chapter about nested structures in SQL. The author presents two possible approaches:
* Parent – Child relationship, basically [node_id, some attributes, parent_id] – easier to maintain, harder to traverse
* Otherwise using enumeration of depth-first traversal, basically [lower and upper bounds for each tree, which gets smaller in sub trees] – harder to maintain, easier to traverse

Plugins / Virtual table

The ability to write plugins rather easily is also one of SQLite special attributes. It allows you to write custom scalar and aggregate functions which can then be used in SELECT queries.

The idea of virtual tables are pretty cool. You can use SQLite with any format you want without importing the underlying data directly. For example, you could write a virtual table to handle logs or csv files.

The only problem I have is that it takes a while to write these and then you could either just dump them into the database or you would use a faster DB for bigger files. So I’m not quite sure when there’s a good application for that.

All in all, the book was an interesting read. I think the biggest payoff is for people working with SQLite and C.

Smalltalk, Objects, and Design by Chamond Liu (1999)

After reading Extreme Programming explained I wanted to learn more about Smalltalk. You may ask why but the whole Agile / XP / OOP stuff grew out of Smalltalk. I heard of it before and knew that it works with a complete integrated development environment which can be saved as images. I think a great concept.

So I looked up some books and this one was recommended. Also I want to learn about some pure OOP. I know most of the basics but most of the time it felt clumsy. However, like often, great concepts get bend over time and you can really see their true nature. That also why I like older books on some topics.

I have a bit of a feeling that Smalltalk is to ‘enterprise’ software as lisp is to AI & co.

I will use Squeak which is a modern implementation of Smalltalk and backwards compatible. Also Squeak by Example is a pretty useful reference for figuring out how Squeak works – it’s also free.

Objects

  Humans as a metaphor for objects, e.g. ‘this account should be clever enough to check the balance’

I’m astonished by how good the book is written so far. Wow. Somehow I feel like I found some ancient tome which was hidden and I can learn from the ancient masters. Or I watched too much Skyrim the last week.

• ‘Objects promote a common vocabulary’
• ‘It is more important to concentrate on clear objects than on the side effects; unless the objects are clearly understood, they will be neither produce nor resuable.’
• ‘class’ as a factory (not really a collection)
• Subclasses have more qualities than the superclass

Smalltalk implementations work with a image, there’s an build-in object browser, a debugger, source control and a testing suite. Incredible if you think about how modern it was years ago. It really feels like the equivalent to lisp but in the enterprise field.

• ‘no one can expect to produce resuable designs without considerable trial and error.’
• Sarkela: ‘Not until you try to reuse do you discover what’s wrong.’

• abstract class = class without any instances

• abstract classes are a repository for expectations
• pure virtual function (C++) / abstract method (Java) / subclassResponsibility (Smalltalk)

CRC Cards

• Class-Responsibility-Collaboration Cards
• Responsibility: What should instances of the class ought to be able to do?
• collaboration: What class uses other classes?
• ‘Better to have accomplish something than nothing at all’ – regarding over-engineering

When (not) to inherit

• Example: Face has a Mouth has a Tongue
• Face inherits from Mouth inherits from Tongue – can do more
• Alternative: mix-ins, e.g. Face inherits from both Mouth and Tongue. Looks like inheritance but is a composition
• ‘Inheriting, thereby tightly coupling two classes, is a long-term commitment’

Tradeoffs

1. Code reduction
2. Subversive superclass message
3. Maintainance

• ‘If in doubt, consider buying’
• ‘In short, multiple inheritance adds power to a programming langauge, but at a cost of complexity.’

Use cases and dynamic relationship

• ‘Testers test what software does’
• ‘for many diagrams the chief benefit comes through the human exchange in which they were created’

Designing the UI

• Designing UIs for games are easier because the underlying metaphor is clear
• Designer has to have a (coherent) conceptual model

Polymorphism

• Polymorphism = several classes with the same method evaluated at runtime
• Substitutability = if it works  for a class it has to work for its subclass

Design patterns

I won’t list the pattern but rather my experience about them. I noticed that I could do a lot of these things easier using functional programming.

‘In the years to following publication of Alexander’s books on patterns [the originator of design patterns in architecture], he realized that patterns alone did not ensure that the results would have “the quality without name.”‘
So I decided to reread Norvig’s presentation about Design patterns. It’s mind-boggling how we ended up with C++ and Java.

Conclusion

Incredible book. I love it. I didn’t want to learn Smalltalk to write Smalltalk but to understand OOP better and especially the paradigm of ‘everything is an object’. This book worked great for that. The author writes superbly and is extremely humble. A great book.

Extreme Programming Explained by Kent Beck (1999)

I learned about Agile and XP the first time from a talk by Volker Birk. When I first heard it I was astonished. One of the most salient things was the focus on the people. I heard about the waterfall model before that it was very process driven. But in his lecture he emphasized how important the team is.

While I worked as a quality manager I started off with the typical literature and it still focuses a bit on the people but – understandably – more on the process. And I think that a good process works better than a bad one. But in fields of design / craft like software development or graphic design I’m now more convinced that people matter a ton.

While reading books about software development on the ‘management’ side I’ve seen over and over again that people matter. I think it’s one of the truths which are just hard to swallow and therefore some people try to ignore it. Coming up with new methodologies, patterns, etc.

That is, I’d rather have my product developed but a small but great team than by a big but average one. Every time.

It goes against the egalitarian idea of ‘everyone is equal’. But there are just differences. You can expect two people two be equal – and especially not given a skill. I couldn’t just join the olympics team for swimming. And even if I trained as hard as them I have some physical problems (e.g. my funnel chest) which just reduces my breath capacity by 30%.

The last few weeks / months feel like catching up where I left. I thought about buying this book in 2009 and now I finally will read through it.

I think it’s an absolutely beautiful book. It’s full of wisdom. However, most of these practices are adopted – even if just partly, today.

XP designed for a team of 2-10 software devs

In contrast to typical software engineering projects with 250+ people.

“How would you program if you had enough time?”

I love this. It focuses on the idea of quality, craftsman ship and being proud of your product. I think this idea can be applied to any commercial creator activity – writing, graphic design, etc.

The driver of a software project is the customer. If the software doesn’t do what they want it to do, you have failed. Of course, they don’t know exactly what the software should do. That’s why software development is like steering, not like getting the car pointed straight down the road. Our job as programmers is to give the customer a steering wheel and give them feedback about exactly where we are on the road.

Constant feedback, customer-driven and design. I love his absolute statement: If you don’t solve the customer’s problem you have failed.

We will be successful when we have a style that celebrates a consistent set of values that serve both human and commercial needs: communication, simplicity, feedback, and courage.

n.c.

Early production needs a little explanation. One of the strategies in the planning process is that the team puts the most valuable stories into production as soon as possible. This gives the programmers concrete feedback about the quality of their decisions and development process that never comes until the rubber meets the road. Some programmers have never lived with a system in production. How can they possibly learn to do a better job?

100% agree. People have to see the product even if it’s just some scribbles on paper. Seeing is believing. Showing that something works or doesn’t work is more convincing that almost every argument.

XP says to do a good job (tests, refactoring, communication) of solving today’s job today and trust your ability to add complexity in the future where you need it.

That’s my modus operandi for years and it works superbly well. Especially if you can deliver a solution which is might do less than some other one but will be more cost-effective. Customers love that.

And why not? Most customers aren’t in the software business but in the X business (where X isn’t software). They want their problems solved.

The difference is between playing to win and playing not to lose. Most software development I see is played not to lose. Lots of paper gets written. Lots of meetings are held. Everyone is trying to develop “by the book,” not because it makes any particular sense, but because they want to be able to say at the end that it wasn’t their fault, they were following the process.

I would argue that that’s in inherently cultural issue. A typical blame culture. It’s drives by applying blame instead of taking responsibility. It’s a culture driven by fear.

Work with people’s instincts, not against them—People like winning. People like learning. People like interacting with other people. People like being part of a team. People like being in control. People like being trusted. People like doing a good job. People like having their software work.

This is one of the main insights I took from this book. Work with the people not against them. I think I quoted that later but Kent Beck says that you need to reward the short-term payoff. You have to work with the ‘natural form’.

Some folks really like this about XP, that it celebrates what programmers seem to do when left to their own devices, with just enough control to keep the whole process on track. One quote I remember was, “XP matches observations of programmers in the wild.”

n.c.

Fortunately, there is a short-term reason to write tests. Programming when you have the tests is more fun than programming when you don’t. You code with so much more confidence.

It doesn’t seem so at first but writing code when you have working tests is beautiful. It’s so much easier. I think Beck wrote this in this book but I can’t remember which one it was. But the basic gist was the the dev got legacy code and wrote the tests first before editing any existing code. I think that approach is pretty awesome. It makes one so much more comfortable writing and editing code.

So you code because if you don’t code, you haven’t done anything. You test because if you don’t test, you don’t know when you are done coding. You listen because if you don’t listen you don’t know what to code or what to test. And you design so you can keep coding and testing and listening indefinitely. That’s it. [This is] called refactoring.

n.c.

If you don’t have a reasonable place to work, your project won’t be successful. The difference between a good space for the team and a bad space for the team is immediate and dramatic.

I found this rather interesting. I didn’t have a read about the influence of work space on team performance so I can’t add that much value. However, I’ve seen some problems myself and I saw how fast a good team can detoriate if people are separated by simply a door.

Do only the planning you need for the next horizon—At any given level of detail, only plan to the next horizon—that is, the next release, the end of the next iteration. This doesn’t mean that you can’t do long-range planning. You can, just not in great detail. You can cover this release in great detail and cover the next five (proposed) releases with a set of bullet items. Such a sketch will not lose much over trying to plan all six releases in detail

n.c.

The person responsible for implementing gets to estimate—If the team takes responsibility for getting something done, they get to say how long it will take. If an individual on the team takes responsibility for getting something done, they get to say how long it will take.

Skin in the game.

 

The next paragraphs are mostly quoted from the book:

Planning Game

• The goal of the game is to maximize the value of software produced by the team. From the value of the software, you have to deduct the cost of its development, and the risk incurred during development.
• Story cards: Task description, estimate, story no., …
• Business & dev working together
• Explore, Commit, Steer
• Exploration: Write a story, estimate it (& split it)
• Commitment: Sort stories by value & risk and then set scope given velocity
• Steering: Business picks scope (one to three weeks worth) and dev deliver – deadline remains the same but scope can change

Development Strategy

• CI: No code sits unintegrated for more than a couple of hours. At the end of every development episode, the code is integrated with the latest release and all the tests must run at 100%.
• Collective Ownership: Collective ownership tends to prevent complex code from entering the system in the first place. If you know that someone else will soon (in a few hours) be looking at what you are writing at the moment, you will think twice before putting in a complexity you can’t immediately justify
• Pair Programming: In my experience, pair programming is more productive than dividing the work between two programmers and then integrating the results. Pair programming is often a sticking point for folks wanting to adapt XP. All I can say is that you should get good at it, then try an iteration where you pair for all production code and another where you program everything solo. Then you can make your own decision.

Design Strategy

• The design strategy in XP is to always have the simplest design that runs the current test suite.

1. Start with a test, so we will know when we are done. We have to do a certain amount of design just to write the test: What are the objects and their visible methods?
2. Design and implement just enough to get that test running. You will have to design enough of the implementation to get this test and all previous tests running.
3. Repeat.
4. If you ever see the chance to make the design simpler, do it.

• For the first iteration, pick a set of simple, basic stories that you expect will force you to create the whole architecture. Then narrow your horizon and implement the stories in the simplest way that can possibly work. At the end of this exercise you will have your architecture. It may not be the architecture you expected, but then you will have learned something.

Testing Strategy

• Here’s what XP testing is like. Every time a programmer writes some code, they think it is going to work. So every time they think some code is going to work, they take that confidence out of the ether and turn it into an artifact that goes into the program. The confidence is there for their own use. And because it is there in the program, everyone else can use that confidence, too.
• You should test things that might break. If code is so simple that it can’t possibly break, and you measure that the code in question doesn’t actually break in practice, then you shouldn’t write a test for it. If I told you to test absolutely everything, pretty soon you would realize that most of the tests you were writing were valueless, and, if you were at all like me, you would stop writing them. “This testing stuff is for the birds.”
• The programmer-written unit tests always run at 100%. If one of the unit tests is broken, no one on the team has a more important job than fixing the tests.
• The customers write tests story-by-story. The question they need to ask themselves is, “What would have to be checked before I would be confident that this story was done?” Each scenario they come up with turns into a test, in this case a functional test.

But there are some absolute showstopper that prevent XP from working—big teams, distrustful customers, technology that doesn’t support graceful change