The Australian military has recruitment problems, especially fit determined individuals who are prepared to sign their lives away for five years at a time.

We hear constantly about the queue of people who want to become Australian citizens.

My theory is why don’t we combine these two ideas and create an Australian Foreign Legion. Provided they serve for the period and don’t misbehave then we can give them citizens, we have well trained determined people who can join our society. People who have proved they really want that as their goal. We would also get a military who has soldiers that can fight for the country without the qualms of sending the country’s own citizens.

I actually think this would work.

What do you think?

I have no doubt that these shows will not be as good as the original.  I think they will be terrible and make me feel worse than I do when a beloved book is turned into a 90 minute screen play. I would like to say that these will probably be short lived but depending on the audience viewer numbers the networks may decide to keep them on indefinitely like dancing with the stars.

This has been done many times before, there was a short lived australian Wipeout last year and even masterchef is an English knock off.  Some shows have managed to use an adapted format and bring it to this country with success but these are based on shows that were not watched here initially and have presented themselves as effectively new content, spicks and specks, letters and numbers and good news week (the initial run) are examples (but none of these kept the original name)

My big question is why do we feel so inferior that we need to buy someone else’s trademark and live off their shirtsleeves just to get a television show made?

Surely we could have ideas of our own that produce similar results without the stigma of always trying to live up to something it is not?

We could have our own car show, with entertaining hosts who are able to review cars and perform stunts and have a good time without having to call it top gear Australia?

Surely we could have a competition between two chefs with a theme ingredient without having to call it kitchen stadium and poorly try to imitate something the program directors at channel 7 clearly don’t understand?

We could even try to be inventive and come up with a format that is not tied to any of these models?  The channels could even sell this on to other countries to adapt.

I am guessing that this all comes down to the reason that TV exists.  As a viewer you want to believe that television is made for people to be entertained, for something to be created that is clever and enjoyable and essentially a piece of art.  The commercial television stations however are not a charity and looking at their owners are not art connoisseurs either.  They are there to make money and don’t pretend otherwise.  I’m guessing that having a brand name of something that people know already is what they are paying for, not for it to be good or creative or even entertaining but to get people to watch.  The more people to watch the more money.  If they did it on their own they may get a better product that is more entertaining and even potentially much cheaper that they could sell on but there are no immediate benefits of pursuing such a strategy.  So therefore we get lumped with content that is laughable not laudable.

Television seems to be becoming more like movies, the content is less important than getting people to watch it – so you either put a star in something contemptible or you buy a brand name and the viewers will come.  Think of the awful remakes of 70′s tv and comic book movies that are common at the moment, they do not do justice to what they represent but that does not matter, its the dollars that count.  Similarly, there seem to be a lot of ensemble movies lately with stars that cover the spectrum of genres but the movies are generally far from the best to put it mildly.

I would like to think there is a solution to this, looking at the sheer number of fine BBC productions potentially we should be investing in an ABC that is able to produce more local content.  I’m not sure however if we have the population to support this. I guess we can either watch sport, things from overseas or crazy youtube dramas like Ikea Heights.

Do you have any ideas for TV shows that should be made?  Had you considered producing them yourself?

photo credit: x-ray delta one

I fully support the ideal and do want to live the way I want to live with the kinds of people that I want to live it with.

The more I think about it the more I think that the current systems we are calling democratic are far from that.

We have the situation where you never meet personally with the elected representatives so they have no idea what you think.   If you write to them and there is no election on then they do not get back to you.  If you elect someone from the opposition your voice may be heard but has no say in what actually happens so half everyone’s votes are wasted.  If you vote for a party based on their policies they may change it in the middle of the term or even a couple of days in if “things have changed” so you actually don’t know what you are voting for when you cast your vote anyway.  Controversial decisions are not even mentioned at elections even though they are planned and implemented later.  I love that if something was briefly mentioned at an election fought on totally different grounds then they have a “mandate” and have to pass that legislation.  If something was not mentioned then it is something that “is in the best interests of the state” and therefore must be passed too.  Finally if something was the opposite then as mentioned before “Things have changed” and it must be passed.

As far as I can see that if you base elections on when you vote then you don’t know what you are getting and your say is basically wasted as it will come down to one of two parties that are pretty much the same in charge.  The people in these parties are ruled by consensus of the party including unelected officials.  As you can see it does not sound like the people being governed by the way they want to be living how they want to live.

If you look at the practicalities of the situation you have people voting in electorates based on where they live.  This ensures that you get a broad mix of people who may have nothing in common voting for people arbitrarily based on location.  This feels like a system geared towards keeping the status quo and not representing my values.  This also leads to the situation where a key group of electorates has more power than others and their interests are overrepresented in federal decisions compared with the rest.  We also get the farce where for a few issues at the moment 80% of the population support having a particular law changed (I can name 3 at the moment without really trying) where they are not being passed because of religious interests which could topple the marginal seats.  Again hardly democracy at work.

I have been giving this a great deal of thought and I feel a lot of these problems is because populations have been increasing whilst the system remains the same.  Comparing Australia now with 100 years ago, the number of representatives has not increased proportionally with the number of seats.  This leads to less representation per person and more farcical situations.  Technology has improved incredibly such that whereas when things started there could be months in gathering public opinion it is now taking days.  I wonder if it would not be best to keep democracy but change the way it is implemented such that people can live the ideal.

I have done a lot of reading recently about small and micro nations.  I feel that they may be one solution.  I don’t see why communities (especially rural ones) can’t create their own countries.  For all intents and purposes solar and wind could power them, they could use rain harvesting for their water and self treat their sewage.  There are enough resources online for them to educate their communities.  I think that I could draft a beginning set of laws for the nascent country and they could have a democratically elected government who truly represents the wishes of the community.  Policing could be achieved.  The only things that I think there would be problems with are Defense, Hospitals and Jails.  I don’t see however why countries could not form cooperatives just like NATO and have a common defense force amongst the micro nations.  For hospitals the countries could pay other countries a realistic price for providing those services to their citizens.  Not every micro-state would need its own hospital.  Finally for jails depending on the laws there is no reason that the vast majority of prisoners could not serve out their sentences in the community.  Using restorative justice why can’t prisoners help repair things or work off their debt to the community.  It is not as if they are not strangers if they are a citizen of the place.  If they are visitors then prisoner sharing arrangements such as those proposed between Indonesia and  Australia they could work between communities.

Alternatively we could change the way officials are elected? If people can chose where they live why can’t they choose the electorate they wish to vote in? I would like to see members who want to represent the wish of service people or healthcare professionals or other areas, with 150 seats there could be a lot of focused politicians supporting a specific aspect of society.  Then the wishes of their members could be clearly communicated.  I think if individual members campaigned for specific platforms specifically targeted at a group of society we would remove the party stalemate that seems to be dominating the way our country is run.

I am thinking of researching the creation of Micro-states and how they could be recognized as part of a Masters / PHd thesis.  What do you think? Do you feel adequately represented or do you feel like other agendas far beyond your control are shaping the way you live?

I can see both sides to why the authorities want to view data and the converse about why it should be kept private.   I will explain why I think this is futile and something not desirable.

Often people suspect that if you encrypt your data then you must be a criminal or a terrorist.  They say that if you don’t have anything to hid then you should not mind handing over your data.  I support encrypting things because of many different reasons;

Reasons for encryption

I am not a paranoid person ~ I know that people read things stored on computers.

• I want to do secure Internet banking and want a secure platform to do this from.
• I know that businesses routinely do things like read people’s email (they claim they can because it is in their IT policy, although I doubt it is that compliant with national privacy principles)
• I know that chat records are routinely intercepted.
• I know that email providers such as google have in the past handed over user names and passwords to governments (which may or may not have jurisdiction over the owner) and I know that mistakes are routinely made.
• From working at a mobile phone company I know that phone records and what happens in phones is subject to requests from agencies who may not necessarily have a warrant to view those records and internal fraud teams who may also Cross reference your data.
• I know that some companies are pushing heavily to be able to seize computers and accuse you of copyright violations without what I would call a high standard of proof.
• I know that you are rarely the only one to use a computer – you may have friends, acquaintances or relatives who want to use it to check facebook and may look for other things even by accident.
• I know that a computer may be stolen and people love personal information for nefarious means.
• I know that after studying the law for over five years that if the police want to get you for something then they will even if it was not what they were originally trying to get you for then they will take whatever else if they are out to get you.

Unreasonableness of nothing to hide argument

Just because you have private data does not mean you want that knowledge made public.  Of course I personally don’t have anything to hide but you may.  You may not want your boss to know that you are looking for another job, you may not want your mother to know what you get up to in your spare time you may not want your partner to know what you are getting them for christmas or you may not want your identity to be stolen and you may not want the police to accuse you of something becasue of a worm inserted on your computer downloaded something dodgy… so why let people know if you don’t want to?

There is no guarantee that if the police or government authorities or anyone else having a look won’t keep your data private even if what they find is not what they were looking for.

Means of encrypting your data

You can encrypt your web traffic using a vpn server to stop it being intercepted

You can encrypt sms messages on an iPhone (http://www.redmondpie.com/send-secret-encrypted-sms-text-messages-from-your-iphone-using-secretsms-app-jailbreak-req/)

You can encrypt your email using many methods including PGP and http://enigmail.mozdev.org/documentation/quickstart-ch3.php.html

You can encrypt your chat records off multiple different protocols using OTR http://www.cypherpunks.ca/otr/

You can encrypt your whole hard drive using truecrypt http://www.truecrypt.org/

There are many alternatives to the above scenarios and other products that produce the same results.   Whenever you go to a website with https you are encrypting data so people in the middle can’t see what is going on, on most websites your passwords should be encrypted so they can’t fall into the wrong hands.  Without encryption many of the benefits of the modern internet could not function.  So even if you are unfamiliar with the technology you are using encryption probably every day.  These are just means to enhance things that y may not be encrypting.

photo credit: CarbonNYC

How Encryption Works

All the encryption methods above are based on the same premise that data is scrambled using a key (a very large number) so only someone with the appropriate key to decrypt (may or may not be the same key that encrypted) can access the data.  To unlock this key you can use a file or password which is then used to produce an even longer stream of data which is used for encryption.  The longer the file or password and the bigger the key the more secure your data should be.

If you use enough bits of encryption (a larger number) and there is sufficient entropy (randomness) in the key theoretically it will take longer than the universe has existed to crack these encrypted files.  For all intents and purposes if it is locked then it can not be unlocked.  I’m sure that in 100 years your data is probably not going to be safe, due to either advances in quantum computers or more likely a breakthrough in computing very large prime numbers.  Some people think that the NSA can crack modern encryption.  If they could, I think it will be like bletchley park in world war 2 and it would be used sparingly and only known by the elite few, I’m sure they will not be pointing their attention to the average reader of this post any time soon to read their compromising emails.  I’m sure whoever discovers it will be stealing money on an unprecedented level if this were the case.

As you can see the key for encryption is very valuable if you want to keep your information secret.  The key can be a password or can be a file or a combination of both.

Should the authorities be able to obtain my key?

If you think that the threat of terrorism is an all encompassing threat that involves giving up most of your personal freedoms then maybe it is acceptable for more of your rights to be trampled and you to be forced to give it up.  If you don’t mind if everyone knows everything you don’t want them to find out then fine (If you really believe that why don’t you post all your personal information on facebook all of the time??).  Looking at the history of recent totalitarian and other oppressive regimes I don’t think having authorities knowing everything about everyone is a particularly good idea.

I read the other day that as we know in most democracies, you are innocent until proven guilty, that there is no onus on you to help prosecution because you can not be more innocent than you already are.  I agree with this sentiment, If there is evidence of a crime on a computer they should be able to find the evidence independently of you having to give up your freedoms.

There is precedent that currently a judge can force someone to give up the keys to their safe else they will force the safe open, this is analogous but not the same because the encryption can not be brute forced.  It is also different because the safe is a physical object and everything in the safe would have been placed there by someone with a key a computer may hold information that was not placed there with consent and the evidence it contains may not reflect a physical occurance taking place.  Besides i’m not that keen the idea that people can order my safe opened but that is another post.

The major reason I’m against the revelation of passwords is because it may be an exercise in futility.  You can’t prove what someone knows and they may not know the key or the password.   The key or password may not exist and even it it does what it reveals may not be what the authorities are after.  The password may only reveal a glimpse of what is on the computer and there may be many levels of encryption which may be impossible for the authorities to detect and therefore not reveal what they were looking for anyway.  Truecrypt for example lets you encrypt data so if you enter one password you can see one set of files, if you enter another one more hidden files will be revealed.  It seems from my research impossible to prove beyond reasonable doubt that the second encryption actually exists.  Different forms of encryption can be combined meaning that the system may be unlocked but innocous looking files can contain extra information.  Additionally the person may not actually know the password, they could have used different techniques to automatically enter the passwords or keys that no longer exist.

I don’t think it is fair to arrest someone and detain them for accessing something that they can not physically unlock and the prosecution can not prove that they have the ability to do so.  If the burden of proof is changed so they are forced to reveal or they are detained regardless if this can be proved then they are no longer innocent until proven guilty and this premise of our legal system is defeated.

If you make it a crime for people to encrypt their data then you are effectively preventing people from using anything secure on their computers including voip, internet banking, connecting to their workplaces and probably even using a chip and pin credit card (that contains onboard encryption).  They also would not be able to watch dvds and would not be able to play on a playstation or watch something through HDMI (depending on the protocol).  This would also mean that people who have data of their own can be committing a crime by obscuring or deleting it (even having a diary written in code would be wrong).  This scenario appears so ludicrous it may actually come true but would be stupid.

So, if you don’t want to give up your encryption and don’t want to go to jail I think your options are limited.  If you are innocent and you can’t prove it (as you would not be able to) then you would go to jail.  If you were not allowed to encrypt this also would be ludicrous.  Even if you comply they could then say there is another level of encryption (even if there is not) and this could continue until you end up serving time.

Can you have encryption but retain the ability prove you are not be able to decrypt?

After thinking about this very heavily there are several things that can be ruled out as options:

• If you were to install something that changes the key on the computer to something you don’t know when you shut it down then this may not work because of a number of reasons including the authorities could seize your computer when turned on and attach an external power supply to keep it running.  Also – if power was just cut then it would just shut down and not actually change something on the disk.
• If you were to generate a key that is stored online then there is enough replication in the internet to mean those looking for your key may be able to grab a cached copy somewhere online.
• If you were to have a program that was to lock out all users whenever so many incorrect passwords were entered this may not work as when they seize your computer they may have taken a full image of it so can restore after each false attempt.
• If you split the key so two people had 1/2 each then both of you could be co opted into delivering the system access.
• If you were to use a key that resided on a flash drive then this could be confiscated from you and then your security is compromised.

The only solution that I can find is that you need to have a key that can be destroyed without your involvement so you can not recover it and you can prove it has been destroyed.  The only practical way I can find to do this is to use a self destructing encrypted USB key. The funitsu tamatebako seems to be the solution to this.  You can instruct it to destroy your files on a number of incorrect attempts or it not being accessed correctly for a particular period of time – such as 2 days.  So if you put your key there and accessed it just before your computer was seized, then exercised your rights to silence then engaged a lawyer it would be longer than 2 days before they could make you divulge your key, by this time the key would be destroyed irrecoverably.  You could then present this and say this is what unlocks it.  It would mean that you would never get back onto your computer but no one else would either.  They still may not take it as proof there are no other methods but would be fairly convincing in any reasonable court that you can not divulge what you have been requested to do and have complied with all their requests.

Personally I won’t be taking it to that extreme, If i were to encrypt something I think I’d just do the plausible denyability thing on truecrypt and encrypt a virtual machine in the hidden container.  I’m yet to see what the direction of this will be in Australia but I’m hoping we can just take our own version of pleading the 5th and hope our elected representatives respect our privacy.

Further Reading:

Self Destructing Flash Drive http://www.ubergizmo.com/15/archives/2010/06/fujitsu_introduces_its_tamatebako_self-destructing_flash_drive.html
Protection against forced revealing of passwords http://www.techburgh.com/2010/01/05/how-to-protect-when-you-are-forced-to-reveal-your-computer-password/
USA forced divulge case http://news.cnet.com/8301-13578_3-10172866-38.html
UK forced divulge law http://www.out-law.com/page-8515
UK forced divulge case http://it.slashdot.org/story/10/10/05/2038219/British-Teen-Jailed-Over-Encryption-Password

WHAT DO YOU THINK?

What is your opinion of all of this? Do you think that we should be able to give up our passwords? Do you think my method of destructing keys would be effective against a determined legal opponent? Where do you see these laws in 10 years?  Please comment below

The terabell blog used to be updated regularly.  I gave up when went on my honeymoon and never got back into it.  The reason that I gave up was that I did not think anyone was reading it and since I knew my positions on various topics I did not feel I was gaining a lot by shouting my knowledge into what felt like an empty room.

I have recently started again because I wanted to use this blog akin to a diary for personal reference and to be a searchable archive of technological issues and debating positions so I could see how I have changed as time has gone by. 

The reason that I did not think that anyone was reading is because there never were any comments on the site.  I never was emailed and no one questioned what I was saying.  It is only this week after getting at least 50 emails about people unsubscribing from the email feeds from the blog did I realise that there were subscribers.  It appears that there were over 200 however that is shrinking, implying not only were people reading but they liked it enough to subscribe.  Searching online there were even posts about people who read the blog before its reinvention (its pretty much the same but looks different and my old server corrupted so I don’t have the old posts as an archive).

I am therefore asking if you are reading this blog or are interested in learning more about something (or hearing less about it) please drop me a comment or an email and let me know.

I appreciate comments and welcome anyone to put forward their opinion

So please let me know what you think of the site, what is happening, where you are from and help me feel a little less crazy.

Hope you have a good day
photo credit: SpecialKRB

Before, I have shown you that you can use the ^ character to match the start of a line, replacing ^ with a value will enter values at the start of the line.  Conversely if you use $ it matches the end of the line, if you use it you can add text to the end of the line.

You can use it in the same way you would use the ^

For example if you have

field1

field2

field3

and you want to put data behind it you can put $ in the replace value (remember to enable regular expressions) and then if you were to replace it with

 as x from table1 

as well as doing a replace of ^ with select distinct

you can then get the following result

select distinct field1 as x from table1

select distinct field2 as x from table1

select distinct field3 as x from table 1

This can be very helpful if you are trying to build a data warehouse from a database you don’t know a lot about and want to make sure the fields are being used – if you have 100 fields this can save you a load of work.

This is nice and efficient but there are more powerful things that you can do,  these relate to matching and reusing text.

If you had the value

exec spTestProc @abc

and you did a find and replace for spTestProc then you would replace this value, with something else like spLogEvent. giving you

exec spLogEvent @abc

however this is not that helpful as you are removing text and it has to have exactly the same things before and after to replace things and get them working.

In a later post I will show you what wildcards and patterns that you can use but for the moment I will show you how to select a whole line regardless of what is in the line so you can put things in before and after

If you wanted to log a series of stored procedures and you knew each line began with exec sp but then the name of the procedure would come next and then arguments afterwards and you wanted to log the whole line you could do the following it would like this before

exec spProcedure1 @abc, @def, @ghi

exec spProcedure2 @zzz

exec spDoingStuff 123, 344

As you can see a normal find and replace could not replace things; if you want this result;

exec spLogProcedure 'exec spProcedure1 @abc, @def, @ghi'

exec spProcedure1 @abc, @def, @ghi

exec spLogProcedure 'exec spProcedure2 @zzz'

exec spProcedure2 @zzz

exec spLogProcedure 'exec spDoingStuff 123, 344'

exec spDoingStuff 123, 344

You would enter in the replace screen that you want to replace the following value

^{[^$]*exec sp[^$]*}

and replace it with

exec spLogProcedure '\1'\n\1

You can put anything that is in the line where the text exec sp is and you will be right to replace the whole line.  Then in the replace with box \1 represents the line so you can put anything before or after in there to get it to work. Also note that \n is a new line.

Hopefully this can save you heaps of time.  Another example where you can replace a whole line that I find useful is with the new merge statement in SQL server 2008, I generally have the source query have the same field names as the target, so then the format that you are after is

target.field1 = source.field1

The below example takes a list of fields (you can get it from using the select wizard and then deleting select and from and replacing [ and ] and ,) If you have problem replacing [ because regular expressions is turned on you can escape it with \ meaning that whenever it sees \] or \[ it will know you are talking about the character and not a pattern.

field1
field2
field3
field4

Then replace in the replace window replace

^{[^$]*}$

with

\t\t\t\ttarget.\1         = source.\1,

The result you get is now

target.field1         = source.field1,
 target.field2         = source.field2,
 target.field3         = source.field3,
 target.field4         = source.field4,

Delete the last comma and you are done.  Again if you are merging more than 10 fields this is very efficient.

This should look like the below image

Replace Whole Lines in SQL

You should be able to see the same characters we have seen before ^ and $ being used for the start and end of the line.  We will explain more next post but this should get you started.

Also see\1 appears again referring to the whole line - technically it refers to everything matching between the first { and its corresponding }

I urge you to give this a go as it should really speed up time consuming monotonous writing of SQL code in Management Studio.  I always find that once I have the idea and work it out then the actual putting it into practice is boring, so the faster you can do it the less boring it is.

If you have any questions or problems please comment below.

photo credit: JohnSeb

Regardless of your affiliation I think that this has raised questions that have not been adequately answered by the media or the members of parliament themselves.

The main question was if it was constitutional to pair the speaker with another parliamentarian so that neither can have a vote.  This would maintain the status quo of the government if there was a pair from both sides of the floor.  Both sides of politics could not agree.  I think that it is constitutional but against the spirit of democracy, the speaker’s vote in the event of a tie would be preserved so in effect the speaker would have two votes – one arising from the pair and the other from the balancing vote.  Having one member of the house with the net effect of having twice as many votes as any other member seems against the spirit of our democracy.  Equally having one member who will effectively never vote means that their position as an advocate for their constituents is null and void as they are volunteering to never vote, again something that I don’t think is correct.

After reading the Brisbane times article below it highlighted that the convention is for the speaker not to vote with the affirmative in any legislation where there is a tie, effectively nullifying their only vote no the matter.  This means that even if parliament operates without a minority setting the speaker’s vote is not ever called upon in a practical sense, again denying the voters of their electorate a voice ~ supposedly the point of having an elected parliamentarian for each electorate.

This has made me wonder;

Should a speaker be a member of parliament at all?

My research has indicated that the speaker has three main functions;

1. To give the casting vote in a tie
2. To maintain order in the parliament
3. To communicate the wishes of the parliament to the Governor General.

According to the PDF below on the role of the speaker they also liaise with other dignitaries and generally be a senior politician.

I feel that if the convention is that they cast no when deciding votes are called upon then this could simply be written into the constitution and not ever get to an elected speaker.  If they are to maintain order, anyone with a knowledge of the standing orders would be able to operate this function – if they were independent of the government or opposition then they may be able to perform this better.  I would argue an independent judge would be in a perfect position to perform this role.  The communication issue may have been critical in the days before fast modern communication however currently this could be peformed by an email of hansard to a clerk.

I feel that if judges can be independent enough to decide constitutional issues and the interpretation of the law, even though they are appointed by the governor general, similar people could be chosen to perform the role of a speaker with little changes to the constitution.  This way we could have proper order in parliament and we could have people voting for someone who is not squandering their votes because they decided to take an administrative position in the parliament.

References:

http://www.aph.gov.au/house/members/speaker_booklet.pdf

http://www.brisbanetimes.com.au/national/consequences-rise-for-missing-a-vote-20100929-15xit.html
photo credit: Hengest

Cycling is a sport that is like cricket, it has long bits that may seem boring but any second could be a key turning point in a long event that comes down to the wire.

I’ve really enjoyed watching on TV this week’s Cycling World championships in Geelong, although there were no australian winners as they generically say at the end of the day “Cycling was the big winner”

The final road race was one of the best one day finishes I have watched this year, the tension of excellent breakaways, the defending champion showing similar tenacitiy he showed last year, the peloton coming out of nowhere to catch everyone and a strong sprint from a proven athlete coming out and delivering victory to a nation who has not seen it before.

I was very impressed and hope that as Australia has shown that it can put on a first class international event and that Australian TV can actually put on a decent coverage of the sport, hopefully siphoning laws can change so they can show more on the alternative digital channels and we can see more cycling like this.

Congratulations to all who competed ~ what a great race in a great place.

* as an aside it was interesting the effect that a lack of microphones gave the sport, It was clear that those in the breakaways were never really sure how far the peloton was away and even at the end Cadel Evans was not aware of how his compatriot Alan Davis went… I’m not sure it is a good thing as it makes the tactics more of a blunt object and takes away some of the strategy I enjoy.

photo credit: renatotarga

Although there are many who think that logging should all be done in SSIS, this is something that I am willing to debate as I am not convinced that SSIS is the correct tool for all environments.

I have been working on logging how long particular things have been taking and putting it into a database.  I could not find anything online and thought that this may be helpful for someone

photo credit: wwarby

Files

I have uploaded the files for this setup

• Build SQL Server Logging Stored Procedure
• Build SQL Server Logging Tables
• Build SQL Server Logging Stored Procedure

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Excerpts from these files are located in the explanation below.

I have released this under a creative commons license so you are free to distribute this and use it in a work environment but I request that if it has saved you time that you donate or at least keep the attribution.

Logging Overview

Logging is a necessity if you are doing something that is time critical.  Many SQL programmers do not think much about how long something will take to the detriment to their users.  Some things are obvious like build processes that take hours, others are less obvious such as building stored procedures to support a web application where millisecond response times are needed to be seamless to the user.  Just try using something that makes you wait for a long time and you will realise why it is important to keep things moving quickly.

If someone comes to you and says that something is not performing as they expect, or you see that something appears to be running slowly you will want to log that so you can set benchmarks and compare.

This is the purpose of this script, it allows you to setup a testing category ie “Testing DW Build Time” or “Checking running multiple web stored procs” and then you can put in events ie “Build Table 1″ and “Execute name lookup”, this will track the individual events and give you consistent formatting so you can evaluate the impact of any changes you have made.  It assumes that only one event is running per category at a time, but you can have as many categories as you want.

Note that if you are not testing something in a production setting you may wish to clear the cache each time before you run these.

Logging Stored Procedure in Action

Below is a sample script that demonstrates how this works.

declare @retval  as int
exec @retval = dbo.spLoggingCategoryStart 'Test Category'
select @retval

exec spLoggingEventStart @retval, 'Test Event 1'
WAITFOR DELAY '00:00:10';
exec spLoggingEventStart @retval, 'Test Event 2'
WAITFOR DELAY '00:00:01';
exec spLoggingEventStart @retval, 'Test Event 3'
WAITFOR DELAY '00:00:00.1';
exec spLoggingEventStart @retval, 'Test Event 4'
WAITFOR DELAY '00:00:00.2';

exec [dbo].[spLoggingCategoryEnd] @retval

select * from EventCategories where EventCategoryKey = @retval
select * from EventLog where EventCategoryKey = @retval

This has three distinct elements, the first is where the logging starts; it declares a variable to store the ID of the event category. This is is assigned by doing

exec @retval = dbo.spLoggingCategoryStart

This needs to be run whenever you want to create a new category. Just make sure that you don’t lose track of the value by recycling the @retval variable, use a different variable for each category.

Behind the scenes this will save the category, the start time, create an event to say the category has been started.

Then you can start logging test events, all you need to do is pass the id of the category and the name of the event. For example

exec spLoggingEventStart @retval, 'Test Event 1'

This will set the previous event to be finished and create a new line for this event.

The delays are just to prove that the times are logged and that there can be different intervals.

Once you have finished with the category then you need to stop logging the category.
This can be achieved by the call

exec [dbo].[spLoggingCategoryEnd] @retval

Behind the scenes this will save the category, the end time for the category and the event, create an event to say the category has finished and calculate the duration in seconds (millisecond precision) for the category and the task.

You can see from the select statements that you can see when each started and finished and their times. If you run it multiple times you can compare and contrast.

Now all you need to do is put in real events in the script and away you go.

How It Works

Tables

There are two tables, they could be joined by foreign key but if you are using this for testing you don’t want that hassle. One stores information about the category and the other the events. The end times and calculations for durations are NULL fields so they don’t have to be populated straight away.

-- Event Categories Table
CREATE TABLE [dbo].[EventCategories](
	[EventCategoryKey] [int] IDENTITY(1,1) NOT NULL,
	[EventCategoryName] [varchar](50) NOT NULL,
	[EventCategoryStartTime] [datetime] NOT NULL,
	[EventCategoryEndTime] [datetime] NULL,
	[EventCategoryDurationSec] [decimal](18, 4) NULL,
 CONSTRAINT [PK_EventCategories] PRIMARY KEY CLUSTERED
(
	[EventCategoryKey] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

-- Event Log Table

CREATE TABLE [dbo].[EventLog](
	[LogKey] [int] IDENTITY(1,1) NOT NULL,
	[EventCategoryKey] [int] NOT NULL,
	[EventName] [varchar](50) NOT NULL,
	[EventStartTime] [datetime] NOT NULL,
	[EventEndTime] [datetime] NULL,
	[EventDurationSec] [decimal](18, 4) NULL,
 CONSTRAINT [PK_EventLog] PRIMARY KEY CLUSTERED
(
	[LogKey] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

Start Category

The procedures are again rather simple.
The starting procedure looks like this

CREATE Procedure [dbo].[spLoggingCategoryStart](@EventCategoryName as varchar(50)) AS
	BEGIN
		-- Stored Procedure to Start Logging a Particular Category
		-- This will return the Key to use for the events in this key

		-- This also creates a start event for the log to make it easier to read
		-- Used to see why processing is taking so long -
		-- Created AB 30 Sep 2010

		Declare @EventCategoryKey as int

		-- Add the Values

		INSERT INTO [EventCategories]
				(
					[EventCategoryName],
					[EventCategoryStartTime]
				)
			 VALUES
				(
					@EventCategoryName,
					CURRENT_TIMESTAMP
				)

		select @EventCategoryKey = @@IDENTITY

		-- Put in a Primary Start Event
		select @EventCategoryName = ltrim(rtrim(left('Start: ' + @EventCategoryName, 50)))

		exec spLoggingEventStart @EventCategoryKey, @EventCategoryName

		RETURN @EventCategoryKey
END

This inserts the values and uses @@Identity to get the most recent value.
The times are based on Current_timestamp that gives the current time.

It makes sure the category name fits for a task and puts start in and gives the same call as you would for a new task.

Log Events

Events are logged by assuming that the category is correct (you could look it up if you like or you could use foreign keys but this was not that important to me)

It adds a new event and then grabs the id for that event. Using that ID it looks for the most recent event for that category and closes it (if it is still open).
Technically this should have been done differently, and ordered by the date as there is no guarantee an identity will be created in order but 99.9% of the time this is the case and you are saving the start times so can recover in this event.

Create Procedure [dbo].[spLoggingEventStart](@EventCategoryKey int, @EventName varchar(50)) as
BEGIN
	-- Stored Procedure to Log a new event
	-- This will mark the previous event for the category closed
	-- This is on the premise that you can have multiple categories with their own events going at once
	-- But each category can only have one event (is not mutli threaded)
	-- So this will start a new entry and close the previous if it exists

	-- Created 30 Sep 2010 AB

	Declare @EventKey as int

	-- Add the new event
	INSERT INTO [EventLog]
		(
			[EventCategoryKey],
			[EventName],
			[EventStartTime]
		)
	values
		(
			@EventCategoryKey, -- <EventCategoryKey, int,>
			@EventName,-- ,<EventName, varchar(50),>
			CURRENT_TIMESTAMP-- ,<Event Start Time, datetime,>
		)

	-- Find the previous event
	select @EventKey = @@IDENTITY
	Set @EventKey =
		(
			Select
				MAX(el.LogKey) EK
			from
				EventLog el
			where
				    el.LogKey < @EventKey
				and el.EventCategoryKey = @EventCategoryKey
		)

	-- Close the previous Event
	update
		[EventLog]
	set
		EventEndTime = CURRENT_TIMESTAMP
	where
		    LogKey = @EventKey

END

Close Category

This is similar to the initial open task but in reverse. It will close the category by putting an end time in.
It will put in an ending task which serves two purposes, it closes the last task and ensures that there are values for the start and end of the category (if you choose not to put anything in the middle)
It needs to get the name of the procedure (easier in code than to hold two variables in the middle of timing)

It then calculates the times – it gets milliseconds and then divides by 1000 to get the seconds, if you used seconds you would not get that precision.

It is dividing by 1000.0 else you are doing integer division and will lose the end times.

CREATE Procedure [dbo].[spLoggingCategoryEnd](@EventCategoryKey as int) AS
	BEGIN
		-- Stored Procedure to End Logging a Particular Category
		-- This will close off any open tasks
		-- Will insert a Final task
		-- Will calculate duration for the events and for the overall category

		-- Created AB 30 Sep 2010

		-- Get the Category Name
		Declare @EventCategoryName as varchar(50)
		Set @EventCategoryName = (Select top 1 ec.EventCategoryName From EventCategories ec where ec.EventCategoryKey = @EventCategoryKey)

		-- Put in a Finish Event
		select @EventCategoryName = ltrim(rtrim(left('Final: ' + @EventCategoryName, 50)))

		-- Put in last event for category (should close previous one)
		exec spLoggingEventStart @EventCategoryKey, @EventCategoryName

		-- Finish off all unclosed events
		update
			[EventLog]
		set
			EventEndTime = CURRENT_TIMESTAMP
		where
				EventEndTime is null
			and EventCategoryKey = @EventCategoryKey

		-- Calculate Times for All Events for the Category
		update
			[EventLog]
		set
			EventDurationSec = DATEDIFF(millisecond, EventStartTime, EventEndTime) / 1000.0
		where
			EventCategoryKey = @EventCategoryKey

		-- Insert End time for the Category in the Category Table
		update
			[EventCategories]
		set
			[EventCategoryEndTime] = CURRENT_TIMESTAMP,
			EventCategoryDurationSec = DATEDIFF(millisecond, EventCategoryStartTime, CURRENT_TIMESTAMP) / 1000.0
		where
			@EventCategoryKey = EventCategoryKey

END

Conclusion

I think this is quite useful as it is easy to implement and is useful to have the scripts in a bookmark so you can put them in when you see the need.

If you have any alternatives or ways to improve or just want to discuss general SQL issues please contact me or leave a comment.

On the weekend I was at a course to become a sports commission accredited club coach.

What I found interesting was that most of the presentation was not about how to improve someone’s performance with specific sporting techniques, that knowledge was assumed.  Instead it more focused on presentation, delivery, organisation and motivation. I guess the theory was that if you know how to effectively coach someone you can ask for help when things don’t appear correct.

One of the biggest things that I took away however was not those skils but one profound thought;

When someone goes target shooting they are always putting in as much effort as they can.  Beginners may actually be putting in more effort than the professionals.

The more I think about this observation the more I agree with it.  If you don’t want to shoot then you dont, if you do you are not happy with not getting a 10 each and every shot.  This goes true for the novices who want to beat their friends to the people in D division right through.  Other sports this may not be true, you may not want to run your hardest each time you go running, you may not want to tackle perfectly each time you play rugby but in shooting (or probably lawn bowls and archery) if you are there to play then you are there to play hard.

I think this has consequences in the way coaching is done, If you are going to put in a lot of effort then you will probably want help, not have help forced upon you.  If you can come to a realisation that makes the effort required less then you will do so until it effects your performance.

It also changes the way that you treat someone who is not doing what you want them to, it may not be a deliberate attempt to subvert your coaching, rather they are trying their hardest and not getting it.

Wouldn’t it be easier in the workplace or elsewhere if you knew everyone was trying their hardest there too?

photo credit: bk1bennett