Post Slackathon Wrap up

It seems an eternity ago now, but just last weekend a very special event occurred: we held the inaugural Stupid-Ideas Powershell Slackathon, where people from around the planet came together to build and share frivolous things with Powershell.

You like numbers? Here's the numbers:

  • 87 people asked to join the Slackathon and were sent slack invitations.
  • 71 of those people answered their invitations and joined the slack site.

On the actual weekend people contributed:

  • 35 separate folders of powershell code that are publicly available for your reading pleasure (detailed below).
  • 45 files within the slack group (snippets, images, etc)

The publicly available contributions came from 14 different people.

One thing that surprised me was that some people didn't talk in slack at all, but quietly contributed really interesting code, which spoke volumes.

There were also 7 "profile.ps1" files that people chose to make available. These are great reading for anyone who lives by the slogan:

Live fast, die young, leave a well maintained profile.ps1 file.

And there are prizes! Prizes, yes!

People seemed to contribute for the sake of contributing, so I don't want to over-emphasize the prizes. Any contribution is a thrill. In the end I went ahead and sent a NimbleText Bundle (NimbleText + NimbleSET) to every one who contributed public code.

But to award the other sponsored prizes, I put all of the contributions (see below) into a spreadsheet and assessed them all on a range of criteria. After much deliberation, here's how the remaining prizes are distributed:

Prize for Slackathon Fever

The 'slackathon fever' prize was awarded for most contributions and highest points total. The prize for this is a copy of Douglas Finke's book: Windows Powershell for Developers (O'Reilly).

I have the envelope here. The winner is...

It's a tie! Two people with the same number of contributions and equivalent overall score:

Prateek Singh and Douglas Finke!

Well, I suspect Doug already has a copy of his own book. But he will now have to send a copy to Prateek as well.

So Wrong It's Right!

The 'so wrong it's right' prize is for misuse of technology. This was hotly contested, and I was torn between a few. I ended up awarding it to a very deserving though simple entrant: Ken Erwin, with his Favorite Drink script.

Unless the team from Chocolatey issue a protest, Ken will receive a Chocolatey Pro license.

The Toppest of the Top!

This is the big one, the toppest of the top prize also brings a Chocolatey Pro License, but on top of that you get a profound sense of shame and a permanent blotch on your resume. The winner of this was Glenn Sarti for 'Ascii Art Conversions'.

Finally, here's a list of all the contributions and who to blame. (If you want your name redacted just send me 15 bitcoin)

BlameTopic
Glenn SartiASCIIArt-Conversions
Doug FinkeAskWolfram
Chris HuntAudioPeakLevelMeter
Glenn SartiBieber
Doug FinkeCentralLimitTheorem
Lee HolmesDefiant
Ken ErwinDevOpsLibrary
Lee HolmesDominos
Doug FinkeExportDataTable
Ken ErwinFavorite-Drink
Prateek SinghFind-UnsecureWIFIConnection
Prateek SinghGet-Celebrity
Prateek SinghGet-Joke
Prateek SinghGet-Nutrient
Doug FinkeGetChange
Doug FinkeGoogleAndBingMaps
Prateek SinghGoogleMaps
Prateek SinghHangman
Justino GarciaInvoke-Chipotle
Glenn SartiInvoke-Yolo
Doug FinkeMaze
Leon BambrickOut-TShirt
Joe BeaudryPosht-ly
Leon BambrickPowerSpell
Brandyn ThorntonRussian-Roulette
Prateek SinghSet-RandomBackground
Chris HuntSingleSampeMajority
Doug FinkeSpellingCorrector
Prateek SinghTest-AdultContent
Ken ErwinTic-Tac-Toe
Leon Bambrickmagic8ball
Doug Finkemoonphase
Leon Bambrickmusic
Paul Lorett Amazonapowerpi

I wanted to write down everything I learnt, but life is just too short to dig into all that.

A few of the salient points...

Floobits is an OK place to allow people to contribute. There's a few confusing things about floobits that we had to explain over and over, now listed here.

A better name (e.g. workshop instead of hackathon), a more diverse group of organizer, and a clear code of conduct up front might've improved the diversity of the event. On gender diversity for example the event was a failure.

Slack worked really well. Avoid the temptation of playing with slack too much... you could lose days doing that if you're not careful.

Cross time-zone issues were terrible, for me in Australia at least. I basically acquired jet lag.

That's all I've got time for.

 

Less than 3 days until the Stupid Ideas Powershell Slackathon.

stupid ideas powershell slackathon logo

What's that?

It's a newly-coined portmanteau of the words Slack and Marathon. It's a lazy, online event, where people from around the world give as much or as little time as they can spare over the course of a weekend, to achieve Stupid And Entirely Un-Noteworthy Things, with any kind of tenuous link to PowerShell.

Why should you join?

So far there are 53 people who've joined the slack channel (and a further 23 who are yet to accept the invitations they requested)

And these 53 people seem to mostly be very clever and knowledgeable bounders, brimming with knowledge on the PowerShell. They're all very keen to help any inexperienced dabblers, so I hope that anyone who hesitated join before will now jump in while they can, knowing that they'll be well supported by a super helpful team.

Enough jibber-jabber Leon, how do I join??

Alternatively you can:

And how long until it starts?

Run this command to find out...

New-TimeSpan -Start (GET-DATE) -End (get-date -Date "2016-06-17T11:00:00Z")

Days              : 2
Hours             : 11
Minutes           : 26

It'll be here in the blink of an eye!

Any further details?

See my previous post on the topic!

There are now confirmed prizes, such as 2 Chocolatey Pro Licenses, A copy of Douglas Finke's book: Windows Powershell for Developers (O'Reilly) and I'll give away a few of my own products for good measure.

Some of the snippets that have been discussed so far are available at our public floobits site.

Here's an example of a Wolfram-Alpha script that Doug Finke has been working on (click to enlarge)...

wolfram alpha from powershell (click to enlarge)

Also -- you can order a slackathon T-Shirt here:

slackathon_tshirt.png
Order-TShirt

 

The Stupid Ideas Powershell Slackathon

stupid ideas powershell slackathon logo

When is it?

Friday 17th June 2016.

How do I join?

What does it cost?

It's free! And there are prizes.

Where is it?

It's an online event. It is everywhere.

What do I need to do?

Think of something simple, fun, funny, stupid, puzzling or plain wrong that you'd like to achieve in powershell. It doesn't need to be humorous, it may just be some recreational programming you've been kicking around in the neglected pipelines of your mind.

On the weekend of June 17-19 2016, there will be people online, at a special instance of "slack", willing to help you achieve your weird ideas.

At the end of the weekend a summary of all participation will be written up, and prizes awarded in a range of fun categories that are yet to be decided. (Some possibilities below)

Why PowerShell?

Powershell is much maligned. Any time I perform a public act of powershell I am met with the meanest comments. I'm attacked by linux users, insulted by 'real programmers', scoffed at by web developers. But I'm willing to fly my freak flag, even if it does have a powershell logo on it, and want to share the joy of powershell with others.

Why Stupid Ideas ?

I've always approached the serious topic of technical learning through the lens of frivolity and play. And I have so many stupid ideas! Surely there are other powershellers who harbor whacky ideas. Surely powershell is not just a corporate tool, whose only lot in life is to quietly get the job done. There must be a place for stupid ideas, even in the world of PowerShell.

Okay, when exactly is it?

TimezoneFromTo
Sydney/Brisbane9pm Friday, 17th Juneapprox. 9pm Sunday, 19th June
Auckland11pm Friday, 17th Juneapprox. 9pm Sunday, 19th June
UTC2016-06-17T11:00:00Zapprox. 2016-06-19T21:00:00Z
London12 noon Friday, 17th Juneapprox. 9pm Sunday, 19th June
New York7am Friday, June 17approx. 9pm Sunday, June 19
California4am Friday, June 17approx. 9pm Sunday, June 19

That's weeks away!

No, it's only...

> New-TimeSpan -Start (GET-DATE) -End (get-date -Date "2016-06-17T11:00:00Z")

Days              : 9
Hours             : 4
Minutes           : 20
Seconds           : 51

What sort of stupid ideas?

It's completely up to you!

Here's some things I think would be fun to work on:

  • out-tshirt

    A cmdlet that directs your text/images to a custom t-shirt design website, and returns a url where that shirt can be purchased. e.g. "hello!" | out-shirt returns a url where you can buy a t-shirt with the word "hello!" on it. dir *.png | out-shirt returns urls to buy t-shirt for each of the pngs in the current folder.

  • dir *.png | Add-Feature -Moustache -Fangs -Glasses -Scar

    Add-Feature is a commandlet that uses OpenCV face detection to find faces in pictures and add features such as Hitler Moustaches, bushy eyebrows and so on. This could have put my 8 year old self completely out of work.

There is a much longer list of suggestions and resources further down. Those two examples were not particular highlights.

You said there was a form to fill out?

Lo, and what a form!

Alternatively you can:

What sort of prizes are on offer?

I was thinking of prize categories such as:

  • Worst idea
  • Most interesting hack
  • So bad it's good!
  • Evil genius.
  • Worst Prompt Function
  • Best One Liner

Confirmed prizes:

I haven't yet organized any other prizes, though I'm confident I can secure other prizes, along the lines of:

  • Lee Holmes' book (Windows PowerShell Cookbook, O'Reilly)
  • Other software, e.g. powershell related tools. (Hmmm, octopus deploy springs to mind)
  • Other books, on the topic or off.
  • An amazon voucher (I would be happy to donate money to this)

You said you had other bad suggestions for stupid ideas?

Some of the new ideas contributed by people inside Slack:

  • A powershell script that automatically sends flowers to your significant other on important days
  • Try and do anything Xiki can do. Perhaps within VS Code.
  • Estimate a person's clothing sizes in two months projected from Fitbit trends
  • Musical programing. Map each char to a note, a-z 0-9 is 36 chars, the standard piano has 12 notes, that's 3 octaves.
  • nCurses like library in powershell
  • Allow powershell to stream to flooty
  • Find a way to activate Cortana commands from PowerShell
  • Some sample pranks and
  • Chipotle ordering script
  • A RandomActOfPizza.ps1: A group of random pizza recipients and random payees, then you spin the wheel and a random person buys another random person a pizza with random toppings.
  • A Rube Goldberg contraption with as many different technologies into a single thing as possible, e.g. Docker, Windows, Vagrant, Chocolatey, ASP.NET Core, etc.
  • Random resume generator
  • A graphviz wrapper
  • And look -- some early progress on cmdlets for displayed the phase of the moon as an emoji.

ps_graph.png

Can I watch and enjoy without participating?

Most certainly! Lurkers, onlookers, observers, auditors, all are welcome.

I hardly know powershell, I should stay away, right?

No no no -- we love you, I love you, you don't need to know anything about powershell. You're very welcome. If anyone makes you feel unwelcome, I will pat them on the shoulder and say, "heyyy, come on, you're better than that. Have a hug." and then they'll remember that they *are* better than that.

I hate powershell, I should stay away, right?

Hate is a strong word. "Fear" maybe? "Have sometimes been frustrated by"? those are more accurate, right? Give powershell another go. Like Vienna, it waits for you.

There's already a brilliant powershell slack team people can join at slack.poshcode.org, why didn't you ask them to host it?

Yes, that is a brilliant slack instance. I'd urge any powershell user to join. They have over 1200 members, and answer many great powershell questions day and night.

But I wanted a small and cosy location, where I could feel free to create a lot of channels, and do a lot of stupid things. It's hard to integrate stupid into a social structure that wasn't built with stupid in mind.

Why are you doing this? Is this all some elaborate plan to promote your product book??

Oh god no. I have heard this accusation, so I want to specifically refute it.

This whole stupid ideas slackathon is -- for me -- a way to procrastinate instead of working on the book. So that's like the opposite of helping with sales of the book. Because if the book isn't finished there can be no sales. And in any case the book is not targeted at powershell developers.

My motivation for having this slackathon, beyond finding new and more interesting ways to procrastinate, is to share the idiocy. As this guy says on his fascinating blog:

"I'd rather live in a world full of eccentric thinkers than one full of unthinking consumers"‚Äč
—Simon Jansen

 

Hosting an infinite number of apps in the cloud for free, on your own domains.

Longer title: Building and hosting an infinite number of scalable secure web apps on custom domains, with no vendor lock, only using skills I already have, for free.

Sometimes I talk myself into the seemingly impossible. While writing a stubborn paragraph of my book (Your First Product... go sign up!), I wanted to know just how cheaply I could host a custom web app in the infamous cloud.

For example, could I do it for free?

Cheap is good, cheap is wonderful, but FREE is magical. If you can host apps for free, then you can host as many apps as you dream up. Whoosh! Bing! Blam! Another thought? Another app!

Once I'd realized what I was attempting to achieve, I put it in tweet form as this:

And promptly gave up on the idea, as it was clearly BONKERS. Or at least I tried to give up on it. My mind kept turning the idea over... there must be a way to have it hosted for zero dollars, without vendor lock in.

Sometimes if you set up a system with a lot of constraints, you see that it's impossible. And you then need to work out which constraint to relax. (This is called "engineering")

So I did what any true engineer does, I used a spreadsheet. I put all the constraints in their own column, and listed different solutions on each row, then checked which constraints they broke.

Anything involving Azure cost too much, and was too likely to lead to vendor lock in. Heroku had a lot going for it, but relied on technology I don't use. GitHubPages had a lot of advantages -- but don't permit any kind of server-side code. At the same moment as I started looking into 'Parse' I heard that it had been shutdown.

Finally I stumbled on an architecture that suited my needs. It was a hybrid, like that mythical beast with the head of a lion and the belly of a zebra, or however those old myths used to run. They were pretty popular back in the day.

Here's what I came up with.

free_cloud_app_hosting.png

That's right. As befits a zero-dollar architecture diagram, I have resorted to the use of Comic Sans.

The front end is static html and javascript, hosted by GitHubPages. GitHubPages are free, and they let you configure your own domains or sub-domains, so for example I could have "GuessAGuid.secretGeek.net" be served from GitHubPages, provided I own the domain (...and have wrestled it back from some pesky Ukrainian). One downside is that the repository has to be public, as private repositories at github still cost a little money, but since this is just the front end of the website, I see no harm in making the code available.

The front end uses javascript (json) to talk to the back-end, which is an asp.net app hosted elsewhere. The back end is hosted by appharbor using their free tier. Because it's the free tier they won't let me have a custom domain, they assign a url such as http://guessaguid.apphb.com/ but that's okay: this is just a back-end which the customer never need see.

AppHarbor is a great way to host the back-end of a site, because I can deploy to it from the commandline, by just pushing to a repository. I don't want the backend code to be public, so I need a private repository. Private repositories at github cost money. For one low monthly price you get unlimited private repositories... but that's not good enough for my constraints, nuh uh. Fortunately, bitbucket gives you unlimited private repositories for a single user, and appharbor integrates just as nicely with bitbucket as they do with github.

So there we have it. GitHubPages front end, appharbor+bitbucket back end. A way to host an unlimited number of small applications, on custom domains, in the cloud, without learning any new tech or getting slugged with fees from anyone.

(One thing I didn't solve was the 'secure' part. I wanted to have the whole thing encrypted end to end, using the mystical glory of Let's Encrypt, a new certificate authority who are ushering in a golden age of https everywhere. The most commonly suggested way to do this is to use cloudflare with a custom domain in front of github pages, though there's also a kloudsec solution. I haven't tried either, so can add nothing of value on the topic. The 'free tier' of AppHarbor includes 'piggyback SSL' which I think might be sufficient for this example.)

And here's the app:

Guess A Guid ← Can you guess a GUID? Try it and see for yourself!

 

How to Left Pad, for real

So someone removed a bunch of their packages from the node package manager, and this in turn broke a lot of other people's software builds.

There have been 1 million articles written (so far) wherein sweaty-fingered coders tie themselves in predictable knots asking:

  • Does this mean NPM is doomed?
  • Does this mean opensource is doomed?
  • Does this mean opensource wins, because it can respond so quickly?
  • Does this mean micro-dependencies are terrible?
  • Can something still be a knee-jerk reaction even if someone specifically says it's not?
  • Does this show that NPM is an evil corp?
  • Doesn't this mean that you should do a trademark check before publishing anything ever?
  • Should you check in your dependencies? Should your dependencies have checked in their dependencies?
  • Has everyone forgotten how to code all of a sudden?
  • It's like everyone has gone crazy! Has everyone gone crazy!?

...and so on. But I don't want to ponder any of that.

I just want to look, very carefully, at the code itself, in the center of this maelstrom....

function leftpad (str, len, ch) {
  str = String(str);
  var i = -1;
  if (!ch && ch !== 0) ch = ' ';
  len = len - str.length;
  while (++i < len) {
    str = ch + str;
  }
  return str;
}

I've had left-padding (and right-padding) functions on my brain lately, as they were added to the most recent release of NimbleText. I was curious if this function behaved the same as mine. 11 lines... what could possibly go wrong?

So I grabbed this implementation and tested its behavior.

I was surprised to see it gave different results to my function!

Specifically -- what does the leftpad function do if you give it a string such as 'HELLO' and ask it to left-pad it to a width of 4 characters (i.e. a length that is smaller than the initial string)?

In NimbleText, to answer this question I asked my customers, who uniformly pointed to the behavior of Oracle's LPAD function.

Oracle's LPAD function, if given a len that is smaller than str, will truncate the result.

e.g.

LPAD("HELLO", 4) returns "HELL".

So that's what I implemented for NimbleText.

But that's not what this function does!

Instead:

leftpad("HELLO", 4) returns "HELLO".

The difference is minor -- but minor things can have dramatic consequences.

For example if someone assumes that leftpad(someString, 10) has an invariant property that it always produce a string that is 10 characters long, they could soon end up with a security vulnerability.

I mentioned this on Twitter and celebrity whitehat hacker 'OJ' responded with:

I wouldn't want a leftpad() function to trim strings

Which I attribute to a latent desire he has to see more and newer vulnerabilities in code (not that there's any foreseeable shortage of vulnerabilities looming otherwise)

...but anyway -- what would you expect from a leftpad("HELLO",4) ?

Should the package manager maintain a running vote, and the people can decide democratically on every question?

Or should there be... I dunno... what's the dirtiest word in software... A standard?

 

Today I Learned

A month ago I saw an article on Hacker News about someone who created a github repo, in which they recorded any interesting and resuable solutions to the problems they encountered.

I started doing the same thing, storing little markdown files inside my 'utils' repo (the private repository where I keep copies of all the little tools I use on every machine, plus my powershell profile etc.)

Pretty quickly I had 100 such files, and I looked into what I could do with this growing knowledge base. I found out about 'gitbook' which is a way of rapidly turning a github repo full of little markdown files into a genuine book.

So here's the product... the free book I wrote without trying to write a book...

today i learned title image

Today I Learned (TIL.secretGeek.net)

(and here's the git repo, error corrections welcome!)

Inside that book, I've written a short article on "getting started with gitbook" so you can do the same thing, including details on how to use gitbook locally (for example for documentation inside the enterprise!)

 

npm is not just for node... npm is for EVERYTHING

npm -- the node package manager -- is a bit of a beast. In fact it's a lot of a beast. it's a crazed, snaggle-toothed snarling monster hell-bent on ruling the entire world and crushing all opposition beneath its giant cloven hoofs.

If you've only heard of npm in connection with node, you may be excused for thinking, as the name implies, that it is just for managing node packages. node is a specific server-side technology, so perhaps that is all that npm cares about?

Not at all. npm cares about everything. npm does everything.

Here is how I thought of it, when I was young and foolish.

Consider these three basic types of package managers:

  1. Machine-level package managers
    These package managers install entire applications. Think chocolatey, apt-get or yum.
  2. Application-level package managers
    These package managers install components an application relies upon during dev/build. Think nuget (or gem, or npm)
  3. Front-end package managers
    These specialize in components for the front end component. Think Bower. (Or nuget in previous asp.net versions)

I thought npm comfortably belonged in that second tier. It's like nuget I thought, but for node. Shrug.

Then I learned about the "-g" (global) parameter. And realized that with npm you can install packages that are available "globally".

So it has crept up into that top tier as well. It can do what chocolatey, apt-get and yum do.

Not a problem I thought... then I took a closer look at Bower. It's an interesting and well-liked package manager for the front end.

But I started hearing more and more about people preferring to use npm for front end packages too. Apparently you combine npm with Browserify and/or dedupe -- and you quickly see it's got the 3rd category covered as well.

Not to worry I thought. npm is just one part of a rich tapestry of tools.

Task runners, for example, are another category of tool used by web developers. There's a growing number of these task runners out there, and two in particular always seem to be vying for supremacy: Grunt and Gulp. But recently I've been hearing more and more about npm scripts.

"npm scripts" are a way of making npm aware of easy to use aliases that point to longer scripts, exposed by a package.json file.

They give you a convenient way to just use npm for everything.

And that's where we've come to now. You can just do every damn thing in npm. I'm kind of hearing an update to Atwood's law, something like:

Everything that can be done in npm will be done in npm. Any thing that can't be done in npm will be replaced by a thing that can.

Maybe npm stands for "node's perfect monster".

Even if you have never used node, and never want to use node, you will still need to use npm.

By January of 2017, all other technologies are being phased out permanently, in order to make room for nothing but pure npm.

Introducing intellisense-like command completion for npm on windows

I say all of this by way of introducing a helpful tool that gives you intellisense-like behavior when using npm on windows.

Doug Finke has written a powershell module that gives you tab-completion when using npm on windows.

In powershell version 5 from an elevated prompt, find and install the NPMTabCompletion module, from the new Powershell gallery, like this:

Find-Module TabExpansionPlusPlus -repository PsGallery | Install-Module -Scope AllUsers
Find-Module NPMTabCompletion -repository PsGallery | Install-Module -Scope AllUsers

(As shown, you first install TabExpansionPlusPlus)

Then, from a freshly opened prompt (that does not need to be elevated), import the module into your session.

Import-Module TabExpansionPlusPlus
Import-Module NPMTabCompletion

(You will also need to either add those commands to your profile, or run them in every powershell session where you want npm tab completion to work.)

This not only gives you intellisense-style completion for npm commands, it also looks in the nearest package.json file to see if there are any commands (i.e. npm scripts) you've specified locally.

Here's an animated gif of it in action:

npm tab completion

I've published a more thorough guide to installing it, here: npm tab completion with powershell. There are troubleshooting notes as well, in case you fall into the same problems I fell into (including the sad news that the time came to uninstall PsGet which has been replaced with PowerShellGet)

(That documentation is at a "Today I Learned" site I've been building out, day by day. There's a lot of stuff there already. I'll blog about how to build a site like it as soon as I get a moment!)

 

Console Is Forever

When I was a wee-little lad, in the 80's, I loved the console.

What a mystical conversation: human talking directly to machine!

When "Windows 95" was released I thought the console was dead.

When I started "working" as a professional developer, I sometimes used the console, but considered it a quirk due to my peculiar upbringing.

As the WindowsTM versions rolled by, I never shrugged off this "quirk". One day "PowerShell" (Monad) arrived — I adopted it, unlike a lot of my colleagues. I used it on and off.

One day, a colleague said "You use the console a lot... are you a linux developer?" Interesting I thought.

Recently I've looked at linux and npm.

I realise now that my "quirk" — my penchant for consoles — is not in "spite" of GUI superiority.

It's a much simpler reason.

2-D GUIs are nice. One day they may be 3D! (That would be fun.)

But consoles... The good old console. Console is forever.

 

The Secret Life of Connection Strings In Oracle! (Oracle DBA's Hate Me.)

I don't know if you've ever had to work with Oracle, but if you have, and if you're primarily a .net developer like me, I wonder if your experience is anything like my own.

The very first thing you do with Oracle, as a developer, is to connect to it. And you soon discover that the way connections are specified in Oracle is batshit insane.

With a SQL Server connection string, there are a couple of quirks*, sure. But essentially, the connection string holds all of the details that are used to connect. Every detail you could possibly want goes into the connection string.

(* Regarding connection strings. I've always wanted to register the site "www.FuckingConnectionStrings.com" which simply redirects to "www.ConnectionStrings.com" but is much much easier to remember.)

In Oracle, a connection string is just the first mirror in a long hallway full of mirrors. A hallway that leads into a maze of mirrors, that takes you to a forest filled with monsters and more mirrors.

We had an issue recently where a connection string that worked perfectly well on 7 different machines did not work on an 8th machine. When we looked into how the connection worked on those first 7 machines, we found it was different on every single one. When we took this knowledge to the 8th machine, we began to feel a terrible sense of foreboding. The further into the issue we looked, the more terrified we became. Eventually the authorities found our skeletal remains at the bottom of a steep ravine clinging to a well thumbed printout of Oracle documentation.

Here is how you usually connect to Oracle:

You follow some simple instructions, and you get some generic error messages. You're not panicking, you're just a little out of your depth. Eventually some oracle DBA scoffs into the room (that's the way they walk, they scoff as they walk) pushes you off your keyboard and does some magic incantations in front of your computer. You keep looking over their shoulder, trying to learn about this strange magic. You see strange registry keys, shocking environment variables, keywords like "tnsnames.ora" and some other disturbing flashes of technical wizard-pokery. Was that LISP!? For a moment you get one sickening glimpse into the infinite pit of sorrow.

After that it just works, and you soon forget how tricky it was, you confine it to a dark corner of your psyche. The rest of your experience with Oracle is never as daunting as that first connection. But by then the Oracle DBA has cemented their role as an irreplaceable technical God.

Well I'm here to blow the lid open on their whole racket.

I'm here to teach you all the things they don't want you to know. I'm going to expose their little shitshow and all of its pathetic quirks.

Get ready.

In an Oracle connection string, the Data Source attribute is used, not for connecting to remote machines, but for searching your local machine. I'll give you an example. If the connection string looked like this:

User Id=scott;Password=tiger;Data Source=bbSales12

What do you think happens with that Data Source value? (bbSales12)

Here's what happens....

The value of Data Source is an Alias. And Oracle wants to Resolve that alias.

According to the documentation, Oracle first looks in "the connection pool" to see if it has already "Resolved" that alias.

Given that, at this point, we haven't started talking to any remote machine, we must be referring to some kind of local connection pool. Probably some kind of in-memory dictionary. Documentation is sketchy of what that is. But let's press on. Ignore that bit.

Assuming we haven't resolved this alias before, then we need to look at tnsnames.ora.

What is tnsnames.ora I hear you ask?

It is the third mirror. It is magic. It is a file that contains enough configuration detail that your machine should be able to connect to the remote machine.

It is likely to be this file that the administrator was futzing with. (Or some other things that will be explained shortly)

First up: how does Oracle find the tnsnames.ora file?

To find this magical file, Oracle relies on an environment variable, TNS_ADMIN. That is the second mirror.

If you don't have this environment variable set, then Oracle will not be able to find the tnsnames.ora file (or the sqlnet.ora file, which will be described later)

With powershell you can look at all your "environment variables" like this:

Get-ChildItem Env:

Get-ChildItem has these two aliases that you might find more comfortable: dir, ls,

So you can type

dir env:

Or

ls env:

And to see value of the TNS_ADMIN environment variable, type $ENV:TNS_ADMIN

Now, assuming you have that variable set, it will show you a path. Follow that path, and look for a file named tnsnames.ora

(Some people will tell you that tnsnames.ora is located in ORACLE HOME\NETWORK\ADMIN. This is confusing advice that you can ignore. More on that later)

Was tnsnames.ora there? Good. Now look inside. It is a text file. So you open it with sublime or vs code, or whatever the cool kids are using at this time of day.

You should find the alias defined via a piece of configuration language like this:

bbSales12=
 (DESCRIPTION= 
  (ADDRESS= (PROTOCOL=tcp)(HOST=bb12Sales-server3)(PORT=1521))
  (CONNECT_DATA= 
     (SERVICE_NAME=sales.us.acme.com)))

What is that language? It looks a bit like lisp, or a bit like JSON. It sounds a lot like Greenspun:

Any sufficiently complicated program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
Greenspun's Tenth Rule

I haven't been able to find any official name for this syntax, other than "the syntax of the .ORA files". It's used by Listener.ora, tnsnames.ora, sqlnet.ora, protocol.ora and tnsnav.ora

(By the way, whitespace is insignificant, but recommended.)

Now -- what if you don't have a tnsnames.ora file? You could of course go ahead and create one, then try and put the right protocol, host, port and servicename into it.

But there's another intriguing possibility.

Way back at the start of the process, inside your connection string, you can embed one of these weird lisp-like strings directly.

Instead of

User Id=scott;Password=tiger;Data Source=bbSales12

You can have:

User Id=scott;Password=tiger;Data Source=
    (DESCRIPTION= 
        (ADDRESS= (PROTOCOL=tcp)(HOST=bb12Sales-server3)(PORT=1521))
        (CONNECT_DATA= 
        (SERVICE_NAME=sales.us.acme.com)))

How's that for a snappy connection string that rolls off the tongue. In official parlance what we've done is add a "connect descriptor" to our connection string.

Now you must be wondering.... what other syntaxes can this Oracle connection string withstand?

The answer is "pretty much all of them."

As an attempt at irony the wise folk of Oracle invented something called "the Easy Connect Naming Method".

This lets you specify a Data Source like this:

//host:[port]/[service_name]

For example:

"user id=scott;password=tiger;data source=//bb12Sales-server3:1521/sales.us.acme.com"

Now that looks much simpler doesn't it? Why am I stating that this is ironic? Because of this little footnote from the documentation:

Prior to using the easy connect naming method, make sure that EZCONNECT is specified by the NAMES.DIRECTORY_PATH parameter in the sqlnet.ora file.

This introduces the "sqlnet.ora" file which is also searched for in the TNS_ADMIN folder.

Inside that file, if you specify

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

...then you'll be able to use "the Easy Connect Naming Method".

If you're brave you can also add LDAP and ONAMES into the mix. But that's outside the scope of this rant.

This EZCONNECT syntax really is a bit easier than the alternatives, so I don't think their attempt at ironic naming was a complete success ;).

Oracle Home.

Now I mentioned that some people will tell you that tnsnames.ora is located in ORACLE HOME\NETWORK\ADMIN

I'll cover what that means, but first I have to tell you, No! tnsnames.ora is not necessarily located in ORACLE HOME\NETWORK\ADMIN. It can be located anywhere! That's what the TNS_ADMIN environment variable is for: for allowing it to be located anywhere, irrespective of where "ORACLE HOME" is. (Your friendly system admins may locate tnsnames.ora on a network drive where it is easier to administer en-masse.)

Now what is this "Oracle Home" business, hmm?

That is the third mirror down the first hallway on the right. It is a dead end. But I will tell you about it anyway.

When (or rather "if, against all common sense") you installed Oracle, it will have been installed into a location that it refers to as the "Oracle Home".

It's possible that there is an environment variable called "ORACLE_HOME". But failing that -- the value of "Oracle Home" may be stored in the registry.

Where in the registry, exactly? Any one of these places my poor dear friend.

HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE
HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\HOMEID
HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\ALL_HOMES
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services

But as long as TNS_ADMIN is set correctly, you don't need to worry about those registry entries, or the absence of an ORACLE_HOME environment variable.

Okay, that's all I wanted to say about Oracle connection strings. In order to cleanse your mind of this horrible mess, I suggest you learn about how .net locates an assembly at runtime or look into error handling with asp.net MVC.

 

The little known beauty of .ensure files

Ever broken the build?

Everyone who uses a revision control system has, at some point, accidentally committed a file that should never have been committed. For example, a ".dll" or a dreaded ".suo" file.

To save us from this entire category of problem, the ".ignore" concept was created, and implemented as a built-in feature of many revision control systems (for example ".gitignore" and ".hgignore")

Similarly: everyone has, at some time, forgotten to "commit" a file that other parts of the code relied upon. Thus, the build has been broken, time and time again. Throughout the world, builds are breaking at this very moment, for the lack of some file that could've so easily been committed.

There is a widespread but little known feature, as simple as the ".ignore" file, that addresses this problem.

If you're not familiar with it, let me tell you about the ".ensure" file.

A ".ensure" file informs the repository of a few basic standards of file hygiene that must be achieved before a push can be issued.

Much like a ".ignore" file, You can copy a ".ensure" from some one else's similar project into your own, and everything will just work.

You can copy a ".ensure" from some one else's similar project into your own, and everything will just work.

For example, if someone else has written a C# based project, where the ".ensure" file says, in effect:

"If there is a '.sln' file in the repo, each referenced project file must also be part of the repo 
"If there is a '.csproj' file in the repo, each referenced file or resource must also be part of the repo

...then you can copy their ".ensure" file into your own C# project, and you will be protected against so many entirely predictable build breakages.

Admittedly, the syntax of a ".ensure" file is a little baroque, cramming up to four "selection" DSL into the one handy file format (regex, xpath, jsonquery and globs).

But most people never write them: they simply copy them from place to place. (Copying files comes naturally to developers at every rung of the ladder.)

And testing the effectiveness of a given ".ensure" file is very easy (you just try to break the build with a missing file). So the community easily converged on a working set of files for any given project type.

The only problem that anyone has with ".ensure" files, is that they are something Leon simply dreamed up, five minutes ago, while doing the dishes. Implementations are yet to exist.

I for one say: go for it.