The Spectator Sport of Vendor Topping

Salesmen representing five different software vendors shuffled nervously into the room, viewing one another with thinly veiled enmity. They'd each received my meeting invite with details of the solution required, and a few choice hints that our budget was in the millions, but didn't realize they'd be pitching at the same time as one another.

The first vendor approached the white-board and nervously drew a simple square.

an_application.png

He explained that his application completely fulfills *all* of the requirements.

The next vendor immediately interrupted and drew a series of boxes next to that box.

a_series_of_applications.png

He explained that his *series* of applications solves not just today's problems but anticipates tomorrow's challenges.

Before he said another word, the next vendor took a pen and drew a box right around all of the existing boxes.

an_application_suite.png

He explained that his solution is not just a series of applications, but a *suite* of application that perform seamlessly together in an orchestrated...

Before he could finish the next vendor had drawn a box under that box.

a_platform_offering.png

He explained that his suite of applications are part of a *platform*. The platform provides *robustness* that the others lack.

Before he could continue the next vendor seized a white-board marker and drew a big box around the outside of all the other boxes.

a_managed_platform.png

He explained that his suite of applications come with a platform that is part of a fully *managed* service, with SLAs and compliance that other vendors can't hope to match.

By this time, all hell had broken loose and the vendors were openly brawling in the middle of the room. Arms, legs, fists and white-board markers were flying everywhere. One vendor was slamming another vendor's head against the floor and another was being forced to eat a whitepaper.

The fighting continued for several minutes before a security guard, Kev, entered the room, asked all of the vendors to leave and then he asked me how I managed to infiltrate the building and the meeting-room booking system, at a company I left five years ago.

My hobby? Physical infiltration. Watching software vendors try to top each other is just an added bonus. Kevin and I wiped the white-board clean for the 3rd time that week.

 

Which Uri Encoding method should i use in C#/.net?

This too is one of the boring "factual" posts. Sorry Lachlan.

I never know which .net uri encoding (or url encoding?) method to use in any given scenario.

So I've built this informative lookup table you can use, whenever you're wondering what sort of encoding to apply to a string that is part of, or all of, a URL. Or URI, url, Url, Uri, or uri.

(This is exactly the sort of fun stuff you need to think about constantly when writing a brilliant and much-loved tool like NimbleText, so that hopefully, sometimes, just sometimes less other people have to worry about it.)

uri_encoding.png

↑ That was a picture, which you can download and print out. ↓ This is a HTML table which you can download, view source and edit.

Character Url.Encode(s) HttpUtility.UrlEncode(s) HttpUtility.UrlPathEncode(s) Uri.EscapeDataString(s) Uri.EscapeUriString(s)
{space} + + %20 %20 %20
{tab} %09 %09 %09 %09 %09
{\r} %0d %0d %0d %0D %0D
{\n} %0a %0a %0a %0A %0A
{\0} %00 %00 %00 %00 %00
~ %7e %7e ~ ~ ~
! ! ! ! ! !
@ %40 %40 @ %40 @
# %23 %23 # %23 #
$ %24 %24 $ %24 $
% %25 %25 % %25 %25
^ %5e %5e ^ %5E %5E
& %26 %26 & %26 &
* * * * * *
( ( ( ( ( (
) ) ) ) ) )
_ _ _ _ _ _
+ %2b %2b + %2B +
{ %7b %7b { %7B %7B
} %7d %7d } %7D %7D
| %7c %7c | %7C %7C
: %3a %3a : %3A :
" %22 %22 " %22 %22
< %3c %3c < %3C %3C
> %3e %3e > %3E %3E
? %3f %3f ? %3F ?
` %60 %60 ` %60 %60
[ %5b %5b [ %5B %5B
] %5d %5d ] %5D %5D
\ %5c %5c \ %5C %5C
; %3b %3b ; %3B ;
' %27 %27 ' ' '
, %2c %2c , %2C ,
. . . . . .
/ %2f %2f / %2F /
' %27 %27 ' ' '

Remember kids: Cool URIs don't change. People do.

 

In search of brutal honesty

As part of the research I'm doing for the book I'm writing (Your First Product) I've been tinkering with google forms, and different ways of scooping ideas out of your customer's head.

To that end, I built this form which I would like you to fill out:

If that didn't work, you can visit it here:

Please provide some honest, anonymous feedback.

I shared the form via twitter a few hours ago, and already have received from some pretty "interesting" feedback. Mostly people seem to give the same sort of feedback they would normally give, but with more swear words than normal.

If you do have any feedback to offer, it is very much appreciated.

 

Writing Your First Mercurial Extension

I wanted to write an extension for mercurial, and after extensive googling, decided that there was no suitable starting point for an absolute beginner like me.

By absolute beginner I mean any person who's never messed with mercurial configuration before and never written any python before. It's strange that such people are not catered for, because most people on the planet have never messed with mercurial configuration before and have never written any python before. Even Shakespeare and Elvis have this particular attribute in common.

(The often cited 'Writing Extensions' only begins to be comprehensible once you've mastered all the concepts involved.)

So here's my absolute beginner's 3 minute guide, pitched at me and possibly you.

To tell mercurial about an extension, you edit your mercurial.ini file (or on non-windows machines, your '.hgrc' file.)

Your mercurial.ini file is located here:

%userprofile%\mercurial.ini

(If you don't have a file at '%userprofile%\mercurial.ini' then create one now.)

(In powershell, you would edit '$env:userprofile\mercurial.ini')

Now look for the extensions section, i.e. the section headed "[extensions]". (And add one if you need to)

And tell it about your extension. Like so:

[extensions]
helloworld =

It looks like we're setting helloworld to nothing. But we're not.

What we've done there is we've said "Hey mercurial, old buddy, if you can find an extension called 'helloworld', then I want you to enable it."

Naturally this isn't going to work, because mercurial won't be able to find just such an extension. But that technique is good enough for all of the built-in extensions (like 'color' and 'fetch').

At this point if we run the command:

hg help extensions

We'll get the result:

*** failed to import extension helloworld: No module named helloworld

(And a lot of other information as well.)

Very well then, let's give mercurial a little more info, so it can find our new extension.

[extensions]
helloworld = C:\Program Files\TortoiseHg\hgext\helloworld.py

That's neat... but there's no such file. Running 'hg help extensions' at this point will, quite predictably result in:

*** failed to import extension helloworld from C:\Program Files\TortoiseHg\hgext\helloworld.py: [Errno 2] No such file or
directory

...and we'll go right ahead and create the file 'helloworld.py'. Let's just create an empty text file in the right place. (You may need admin rights to create a file in that location.)

Now when we type hg help extensions we'll see the details of all the extensions, and in the section titled 'enabled extensions' it should say:

   helloworld   (no help text available)

Fine. So the next step is to add some help text to our extension.

Go to the empty file and add a 'doc string'. You do this by enclosing a statement in triple-quotes, like this """

"""hello world is a very simple extension
"""

(Python is white-space sensitive, so make sure there's no whitespace before any of those triple quotes)

Now we have our simple extension that does nothing, it just exists and broadcasts its existence.

When we type 'hg help extensions', we'll see the following in the 'enabled extensions' section:

  helloworld    helloworld is a very simple extension

And, even better, if you type:

hg help helloworld

you'll see...

C:\>hg help helloworld
helloworld extension - helloworld is a very simple extension

no commands defined

Ah ha! No commands defined? So our new mission is to create a "command" within our extension!

Here's a very simple command called 'hello'. And it introduces a few other concepts I'll describe in a moment.

"""helloworld is a very simple extension
"""
from mercurial import commands, extensions, util

def hello(ui):
    """say a big hello.
    """
    ui.write('well hello there world.\n\n')

cmdtable = {'hello':(hello,[],'hg hello says hello')}

commands.norepo += ' hello'

In order to tell mercurial about our command, I had to create a list called 'cmdtable' and put my command, 'hello' in it, plus a few details about ways to call it.

I also needed to tell mercurial that this particular command should work just fine even if it is called when there is no repository present. (Thus, commands.norepo += ' hello')

So what have we achieved so far?

now i can type:

hg help extensions

-- and see that my extension exists. and i can type

hg help helloworld

-- and see help for my extension, including details of its commands.

And i can type

hg help hello

-- and get help on its one command.

Finally -- and most importantly, I can type:

hg hello

and have it run my new command.

Now there was a little hint given when I typed hg help hello. The message mercurial gave me was:

list of commands:

 hello    say a big hello.

use "hg -v help helloworld" to show builtin aliases and global options

And that just about wraps it up for a complete beginner. We've reached the hello world stage.

Notice the enticing message mentions builtin aliases and global options. That's what you can look into next, if you really want to achieve some Next Level Mastery of Mercurial Extenions. But for now we're done.

Further reading

 

You Feel The Blood Drain From Your Face (A DevOps story)

I just had a flashback that might be worth sharing.

Over 10 years ago. I was working as a contractor at the head office of a company that had 300 field offices. Every office had its own database. Same schema, same tables, different data.

To help people in head office query the field offices, I'd built "Multi-Query". It was a fairly simple VB.net app, basically a clone of SQL Query Analyzer but with a checked listbox that let you select which field offices (plural) you wanted to execute the query against. Hence you could run a single query against 1 or more databases, as many as 300. The results were accumulated and tabulated for you.

I'd initially built "Multi-Query" for myself, but it was useful enough that some of the other analysts asked for a copy. We were under the pump (always) and this tool would let us get the results we needed. But it was extremely rough, ugly, and error-prone. Back then we'd call it a "foot-gun". Today we'd call it DevOps.

multi_query.png

Some of the other analysts were not particularly versed in writing SQL queries, so I also did a little mentoring about joins and NULL handling and the like.

One day, I'm sitting there working hard, when I heard one of them say "Uh-oh."

At this point I was not alarmed. I sat across from this person (no cubicle divider, just a desk that used to be a lunch table).

Then I heard him ask a question. The words seemed to echo, like they were floating to me in a dream.

"Is there a way that you can put a query into a transaction, and roll it back, after you've run a delete?"

Echo. Echo. You Feel The Blood Drain From Your Face.

"Is there a way that you can put a query into a transaction, and roll it back, after you've run a delete?"

"Is there
     a way
      that you
       can put
         a query
         into
        a transaction,
      and roll
     (roll)
    (roll)

      it back,
     (back)
    (back)

   after
    you've run
    (run)
  (run)

    a delete?"
  (delete)
 (delete)

"After?" I said. "After you've?" My mouth was dry. I stood up. My legs were like jelly. I walked around the desk, a journey of five steps that seemed to take forever. 300 offices. A single delete query. 300 offices. Delete queries run by people who don't know how databases work. The potential for devastation was limitless.

"What. Did. You. Do? (Do) (Do)"

But no. It wasn't you. It was me.

I put the tools in your hands. I gave you the weapon. I handed you the ammunition.

"What did I do?"

Queries don't delete databases, people do.

Lucky for me: it was nothing. It was a single row of config data in a single office.

It was NOTHING. But it was also a hell of a lucky wakeup call.

The new version, issued minutes later, refused to run any query that contained a "delete" or a "truncate" or a "drop" and it made a noise if you tried. (Not a perfect defense, but a start)

What mattered most was that we took control of a test environment, used it to check our logic first. I needed to lead by example, not lead by jangling my spurs and rushing fearlessly into production every 5 minutes.

Slow down. Be careful out there.

 

UPDATE a table using a JOIN to another table (or to itself)

Soooooooooo, this is going to be one of those boring SQL-Server posts.

Despite writing T-SQL day in day out since forever, I often forget the syntax for an UPDATE statement that JOINS with another table (possibly itself). So here it is:

UPDATE
    p
SET
    p.ManagerEmail = m.Email
FROM
    Person p
INNER JOIN
    Person m
ON
    p.ManagerID = m.ID

Points to remember are:

The first part, 'UPDATE X' is simply 'UPDATE' followed by the alias of the table (you don't need to say the table's name there)

And (contrary to what some internet randos will tell you) you don't need to add a where clause to stop the update from applying to all rows of the table. It will only apply to the rows which are matched by the join condition used. (For an inner join at least ;-) )

I was using TimeSnapper to play back my work when I saw myself struggling with this query. And I remembered seeing myself struggle with the exact same thing another time (also via playback). So I'm putting this here, so that the act of typing it out might help cement the entire thing in my mind.

Also, I find it amusing that if you to do this in MySQL you basically write all the same things in a completely different order. So I'll include a MySQL example here for reference, too.

UPDATE
    Person p
INNER JOIN
    Person m
ON
    p.ManagerID = m.ID
SET
    p.ManagerEmail = m.Email
 

The Zen of Telling Notepad++ That .Config Files are XML

There's a zen beauty in improving the configuration of your tools. Breathe in. Configure. Breathe out. Reconfigure.

Problem: When I open a web.config file with Notepad++ it doesn't realize that it's an XML file. Thus, it doesn't syntax highlight the file accordingly.

A moment of googling led me to an answer at superuser.

Solution:

  1. Go to Settings > Style Configurator...
  2. Under Language, scroll down and highlight "XML"
  3. Beneath the Language list, find User ext. textbox and add "config" to the list.
  4. Press Save & Close
  5. You will need to close and reopen the file, for the syntax highlighting to take effect.

This is one of those minor frustrations that I put up with, year in year out, and don't even realize I'm tolerating it. Why? because every time I edit a web.config file I'm absorbed in the momentary struggle, struggling under the cognitive load of a deep mental stack of problems. I don't have the time (or the mental resources) to stop and fix this problem before getting the job itself done.

Slow down. Get the tooling right. Sharpen the axe.

Spend five minutes to save five seconds*.

It will pay off in time.


(There's also something nicely recursive about configuring the way config is displayed)


footnotes

* The idea of 'Spend five minutes to save five seconds' is basically a variation on:

spend two hours to save five minutes

...which is from the Unusual Time Management Ideas Episode of the 3 Month Vacation podcast. A really interesting show!

 

How Businesses Are Actually Structured.

There's two particular observations I want to share about the way businesses are structured.

In the ideal scenario, a business with N employees can be visualized as a single Directed Acyclic Graph (DAG) with N nodes. [A Directed Acyclic Graph is similar to a Tree... but each connection is directed and a node can connect to more than one parent. So I'm going to say DAG instead of Tree, but you can picture a Tree if it's more natural to you.] So.. I was saying... a business can allegedly be represented by a single canonical DAG. A, B and C report to D; D reports to the CEO.

Well, here's my first observation: Each of the "corporate services" have their own "Canonical DAG".

"Corporate services" are those parts of a company that cost money but (hopefully) keep the other parts of the company functioning, i.e. Human Resources, IT Services, Finance, Building services.

Human Resources will publish one (or many) Org Charts. To them, this is The Canonical DAG. This is the true structure of the company.

The IT department maintain a very complex Active Directory, with users, groups and nested groups. Forget the Org Chart, to them *this* is The Canonical DAG. Active Directory looks at the Org Chart and says, "That's Cute".

The Finance Department have a book of accounts, identified by cost codes, with different cost codes "rolling up" into higher cost codes. To them, *this* is The Canonical DAG. The Finance department look at the Org Chart and Active Directory and say "That's Cute," while fanning themselves with a wad of notes.

Building services divide the corporation into buildings, buildings have floors, floors have workstations and workstations have various resources (broken chairs, broken projectors, broken people, broken printers, broken monitors). To them, *this* is the Canonical DAG.

And so on for every other Corporate Service.

Now, none of these DAG's are as simple as you'd want them to be. There's orphaned branches. There's dotted lines (multiple parents to a single node). There's time-dependent edges (Jill reports to Amy on Monday and Wednesday and to Fred on other days). And a whole lot of other complex modifications*. But it gets a lot worse.

Here's the second observation. The real structure of the company isn't stored in any of these systems.

The real structure is the friendships, rivalries, favors, grudges and slights that are accumulated over years. The social/antisocial network of associations that isn't written down, and makes all the difference in whether an idea will succeed, a directive will be followed, a permission will be granted, a bonus given.

And this network of associations is uniquely perceived by each node within the graph. (And every node within the network will strongly deny that they themselves have any rivalries, grudges etc... this is a crucial part of the way power structures are enforced.) (You might think you can "detect" these structures by looking at who calls who, who emails who, and by detecting tone etc., but the real stuff is never written down or even hinted in writing: that's what gives it power).

Note that the relationship between A and B is different, depending on whether you're looking from the point of view of A or B.

So for a graph with N nodes (people, nodes are people) there are N graphs (most of which have less than N nodes, thankfully).

So, yeh, good luck with that restructure. I'm sure this new (centralization|decentralization) will make a world of difference.

There's a book called Moral Mazes that exposes how companies really work. It's quite dry and academic, but with more twists and betrayals than Game of Thrones. Recommended.


* "Job sharing" is a particularly funny business. If Bill and Bob share a role, we can't represent it as a non-directed relationship "Bill shares in a symmetric manner with Bob" (as our graph is directed), nor can we represent it by two separate and directed relationships "Bill shares with Bob, and Bobs shares with Bill" -- as this is cyclic and makes computers blow up, so instead we have to introduce the concept of roles: "Bill has a role, and Bob has the same role". But now we've made every single node in the entire organization at once more flexible and much much more complex. Good luck with that. We can instead create a virtual person, BillBob, and both Bill and Bob report to the virtual BillBob, who reports to whoever Bill and Bob used to report to. That localizes the complexity somewhat, but also confuses the living hell out of building services and payroll, who assign BillBob a desk and a paycheck, which somehow ends up in the bank account of 'Directed Acyclic Graphs Are Hard, Incorporated.'

(This article has been in my backlog for five years now! Wow. Every single time I've gone to publish it there has been some restructure happening in a company I work with, and I've avoided publishing it for fear it could be interpreted as relating to a current restructure. So I finally thought I'd just publish it with this note saying "No, this does not relate to current events. This relates to eternal events.")

(See also: 3 differences between 'Small Business' and 'Enterprise')

See also: hugh mcleod's perpetual org chart, a classic of the genre, below, and this indepth theory: The Gervais Principle, Or The Office According to "The Office":

sociopaths rule the clueless who rule the losers

 

Launching the secretGeek Wiki

This might be the mid-life crisis talking but lately I've been feeling that I want to do more. More creative work. More of everything. I want to write books, compose music, draw pictures, make movies, create amazing programs, solve impossible mathematical problems, learn magic tricks, create European-style boardgames, print circuits, invent incredible devices, cook, grow, share, give and give.

I've even made plans to bring back the zine. This would involve sneaking into an actual school to use their actual school photocopier to make copies. But more on that another day.

I think it started with the hexaflexagons: just a throw away word, mentioned in an article about Cellular Automata. No: something came before that... I was looking into the possibility of Optical Lego Recognition. OLR! Research on OLR led to image-processing, led to Cellular Automata, led to Conway's Game of Life. Game of life! GOL led to an interview with John Conway who mentioned Martin Gardner and the magical word "hexaflexagons", which he didn't explain at all. How cruel! To mention a word like that with no explanation! Researching hexaflexagons led to Vi Hart, which led back to Martin Gardner. Martin Bloody Gardner! Martin wrote a "Mathematical Games" column in Scientific American, from 1956 to 1981. I'd never heard of him: but he influenced practically everyone that ever influenced me. He popularized all of the topics above and many more. So I fell in. Tumbling like Alice into the rabbit hole. Deep into the wormhole of recreational mathematics, and soon wiki.secretGeek.net was born.

It's a place where I (or we) can maintain a more permanent playground, less ephemeral than blog posts and tweets. A place for documenting certain things. Not things that have any obvious economic value. Things I (or we) find interesting. Interesting things that remain interesting even as time passes.

I want to be able to put together informative, interactive pieces; a little like Red Blob Games, a little like 'the sierpinski triangle page to end most sierpinski triangle pages' a little like Bret Victor's Learnable Programming with a splash of _why's tryruby and a taste of Bob Nystrom's Dungeon generation article (more on that another time).

So I picked up the blog engine from secretGeek.net, tore out its heart and replaced it with a simple markdown-based wiki engine. To make the site interactive, I wanted to create a simple plugin system. I already had a facility for categorizing the articles, i.e. 'tagging' them, so I extended this: each category that is added to a page means a custom partial with that name is included in the page. So if you've tagged an article as being about 'mp3' then the 'mp3' partial would be included if it exists (which it doesn't) and it would presumably allow you to play the mp3s in that article. More generally speaking: tags allow custom javascript to run, that does something special to certain elements within an article.

The first 'custom partial' I added was for 'logo' as in 'turtle logo' that wonderful learning environment from the happiest moments of your childhood. Now I can include runnable logo programs within the wiki, just by adding a blockquote with the class 'logo'. This let me write about a lot of Fractals, Space-Filling Curves and the like.

The next plugin I added was for 'Conway's Game of Life' (gol), so that runnable examples of Game of Life patterns can be included inside articles, for example about Spaceships and Guns.

I also added a plugin for displaying mathematical equations, using mathjax though I've only used that at one article so far.

I've been reluctant to 'launch' the damn thing, feeling I ought to get it 'right' before I do. But then the voice of Amy Hoy materialized in my head and announced 'just fucking ship', so that's what I'm going to do.

Here's a decent starting list of articles:

Over the next 10 years or so, I (or we) can add info about a bunch of other topics as well. For now I've disabled the ability for new contributors to register. But I plan to enable that when I have the kinks removed. This will exist for a long time to come.

Visit the secretGeek Wiki

And since you're still here. Two somewhat relevant images that strike a nerve. A fractal Tom Selleck. And a quote from Barton Fink.

 

How can we do better?

—So we did two releases in quick succession. Why?

—The second release was to fix a typo in the first release.

—How can we do better?

—The release shouldn't have gone out until it had been tested in a QA environment and signed off.

—But couldn't we have caught the error sooner than that?

—We need a build server, that builds and tests everything. A unit test should've picked it up.

—Surely we could've caught it sooner than that?

—The code should never have been pushed to the build server. A code review should be required prior to check in. A second pair of eyes, looking at the diff would've caught this.

—But couldn't we have caught it sooner than that?

—If we were writing tests first, then the faulty code would've been detected before the engineer considered asking for a review.

—But couldn't we have caught it sooner than that?

—If the language we were using had a more expressive type system, it would've been impossible for such a typographic error to compile.

—Hmmm. But isn't compile time too late anyway?

—If the engineer had thought more carefully before pressing each key, been more intentional in her work, this would never had happened!

—Is that really the best we can do!?

—If we selectively bred perfect humans, incapable of incorrect thoughts, then our releases would always be perfect.

—No, selective breeding takes too long. We need to do better!

—We need to manipulate the genes before the zygote is allowed to split and multiply, and perfectly engineer the perfect engineer.

—Ah yes, project Ultra-Mega. Do we have a status report on that project?

—Yes. There were two releases this week, in quick succession. The second release was to fix a typo in the first release.

—How can we do better?

(The meeting took *forever*)