Announcements regarding site changes, updates and other such malakeis
OK. So far nothing has blown up yet. That’s some sort of record. The new setup has such fast page-load times that there’s almost no point in going the last step (moving static assets to a CDN).
In case you haven’t been paying attention to the ongoing spasms of pure awesomeness that have been emanating from MentatSpace recently, here’s a recap.
Dismayed at the incompetence of our webhosts for the domain, your Beloved GT decided to test just how much load he was applying to their bank of state-of-the-art 1980’s desktops.
So he set up an Ubuntu-64 instance on Amazon’s AWS EC2, wherein he replicated everything.
Time taken: about half an hour. Pain: zero. Cost (first year): zero. Yes – zero cost for the first year (provided the traffic remains below about 15Gb… that’s 5x current traffic, and each extra Gb above that costs 20¢/month). [Correction @ 2:17pm: it’s 9¢/Gb]
The instance was deliberately set up to be space- and RAM-constrained relative to the setup at the webhost (which is notionally 1GB RAM and unlimited storage). So the EC2 instance has 8Gb storage, and 618M RAM, and is on an Amazon server in Japan.
Your Beloved GT also replicated the entire MentatSpace on a spare Compaq notebook that is connected to the telly (so we can watch internet video and so forth on the big screen); the notebook had 100Gb free space and 2Gb RAM, but is also running normal computer-y stuff all the time.
The aim of this exercise – once everything was set up, the plan was to kick both machines in the guts and see how much CPU and RAM gets used when, say, the big OzRant cron job is running.
Now the big OzRant job does the following:
- fetches and stores the days’ ecodata outcomes, and compares them to (stored) consensus;
- fetches and stores OHLCV data for every stock listed on the ASX, every industry and ‘TopX’ index, plus indices for NZ, Japan, Korea etc;
- calculates and stores percentage changes, technicals (CCI, %R, and 3 different MAs), and advance-decline numbers;
- updates weekly databases;
- generates ALL the text you see in the ‘vanilla’ OzRant – the tables, the waffle – the lot;
- posts the gentext to the MentatSpace.
It then goes and does some other, time-insensitive tasks (i.e., stuff that doesn’t need to be out as quickly as possible).
So the verdict? The entire cron batch – which is 17 individual PHP scripts, called consecutively by a bash script – finishes in 3 minutes on the EC2 instance and never gets above 50% CPU or 45% RAM. It finishes in the same time on the home media notebook, with lower resource use.
If I wasn’t so stingy about waiting for free (20-minute delayed) ASX data, I could produce the entire schemozzle at 4:18 every day (you might have thought 4:03p.m., but you would be wrong: the settlement time for the ASX is 4:15p.m., and things sometimes change after the close but before settlement). If I had fewer scruples I could do it at 4:03p.m. and tell you it was right, then surreptitiously update it later like the big boys.
Now here’s the thing: not only does the cron job happen 5 times faster on the AMZN free layer than it does on the webhost – the actual site page load times for the actual final Steaming Grogans of Wisdom are faster by a similar magnitude.
And yet it gets better. If some addition to the MentatSpace process resulted in a much much higher resource use – say a new big data-backfill job that involved getting even more financial report data than we already have (10+ years for 1700+ stocks not enough?) – the AWS makes it trivial to bid for a ‘spot instance’. This is effectively uprating the entire machine-instance by a factor of 5 or 6, for as many or as few minutes as you like. Cost: about 20¢/hour.
[Correction @2:14pm… just checked pricing. 20¢/hour is the price for an ‘Extra Large’ spot instance, which would uprate the instance by a factor of more than 15; a 5-factor increase is would be a spot-instance of ‘small’ which costs 2.7¢/hour]
So if I had a massive job that would make the current AWS instance redline, I could – with a one-line entry on the command line – uprate the machine for as long as it took to clear the job.
And that ‘static assets to a CDN’ stuff I mentioned above? Well guess what – the AWS has its own built-in CDN (called ‘CloudFont’). It’s a matter of defining a ‘bucket’ and letting the cache shift the static assets. We can pull the trigger on that shit anytime, yo.
Forecast: anybody who sells webspace is doomed. Smart people will write free software that does all the customisation for you (that already exists for widely-used things like WordPress).
So the average semi-savvy webmaster/blogger will be able to break the shackles of hared hosting, without forking out $30 a month for an uprated sharedhost or $150 a month for a standalone bare-metal rack. (A bare-metal rack would be sweet, but WAY overkill, for MarketMentat in its current manifestation).
I know, right?
You’re probably feeling a little light-headed from the rapidity with which the page appeared.
You typed in ‘marketmentat.com’ and whooshka!, you were spirited to MentatSpace V2.0 in the blink of an eye.
That shit is what we web-genii call a ‘redirect’ (aka a 301).
I’ve put a redirect in place while I’m making sure that the server side is optimised to within an inch of its life; I am root on this beeyatch, and so I can do stuff that I could not previously do (e.g., FILE LOAD LOCAL INFILE in mysql, and some other ginchy stuff that will speed up the algorithm).
In fact not only can I do stuff, I can sudo stuff. Process that, ladies and jenner-menz.
Once the server is set up ‘just so’, I will formalise the attachment of the domain name to the new IP address.
There’s a downside though: the algorithm that produces the gentext bits of each SGW (Steaming Grogan of Wisdom) includes a lot of domain-specific references, and produces images and <a> tags that use the domain name. Left untended, that will introduce negative enhancements because pages will try to grab images from the old MMSpace, and will then be redirected back here – the risk of ever-decreasing circles arises, and we all know how that ends (it ends with the algorithm disappearing up its own bumhole).
In all likelihood I will get the algorithm edited and so on before the market closes tomorrow (Oz time); if not, I will hand-craft tomorrow’s OzRant (today’s OzRant failed to materialise – that is the proximate cause of my decision to dispense as quickly as possible with the old, failed, webhost).
Just as an aside – there are some known issues that need to be fixed tomorrow: the fact that links to individual posts, pages and category archives result in a 404 (and the 404 is not the canonical 404 page) is the first cab off the rank.
As mentioned above, the decision to finalise the move was made abruptly today, and was necessary because of the exponential growth in the incompetence of the former webhosts – it appeared that they had somehow struck a rich vein of either Moronium or Stupidite, and remaining under their aegis was simply dooming oneself to a Stone Age version of the internet.
Suck it up for a day or so – the archive will be better than it was in the old digs, for I am root. (Did you notice I resisted the temptation to say anything about how I am a good root? I should think that goes without saying.)
OK, let's start with a "known known".
Your Beloved GT is one of those oddballs who thinks that #Twitter is reasonably awesome.
From there, some things are natural corollaries.
If Twitter is awesome, it makes sense to integrate it with the bloggy droppings to the greatest extent possible.
In the past, each blog-dropping was automatically blasted into the Twitterverse with a quasi-customised tweet – it appeared in the timeline of @MarketMentat but the tweets were not fully customised because the #hashtags at the end of the tweet did not change from post to post.
Anyhooo… if the plugin that was just enabled functions as it says on the tin, the twitter usernames and hashtags should be automagically linked to Twitter.
This ought to do a bunch of things – things that will be explained at a later date, in a later post. Dark, scary things.
Now, you know that your Beloved GT likes to swim against the current; to boldly declare "Fuck you, asshats" and position his handsome self on the "wrong" side of every bet going.
Guess what the "wrong" side of the Obama bet is? Well, consider what all the talking assholes (CNBC, CNN, Fox, ABCNBS) are babbling about.
They breathlessly repeat the "all options are on the table" mantra – the one currently being trotted out by Hitlerly Clit-ton and the rest of the insane whackjobs in the Rastus Administration.
Does that mean that they – and by extension, the Mass Idiot (their only demographic nowadays) – actually think that the US is going to invade Persia?
No. Not on your life.
In fact, the administration of Rastus HopeyChangey is shitting bricks about ever having to follow through on their Boltonesque hubris. (By Boltonesque, I don't mean "syrupy shit per the collected works of Michael Bolton"; I mean 'foaming at the mouth insane shit, per the collected works of John 'Can I Watch While You Bang My Wife' Bolton").
You see, as I wrote some years ago, Iran has excellent anti-shipping missiles. And while Yank Admirals (and Admirals everywhere) are besotted with big floaty things, those big floaty things actually shitful at avoiding being cut in half.
Sure, you can launch all manner of child-killing "platforms" from an aircraft carrier – but at the end of the day it is a huge target, and in the confined spaces of the Gulf they are fucking DOOMED if the enemy decides to play hard-ball.
So the Rastus Administration are trying to jawbone the Iranians into being good little semi-darkies – make all the right noises, kiss up to the lunatic tribal whackballs of the cock-slitting cult in "Israel"… generally be good little niggers.
I think the Iranians know that the Yanks are just trash-talking: they also know that the Yanks are so fucked after getting ass-reamed in Afghanistan and Iraq, that they don't have the stomach for a ground fight. If anything kicked off against Iran, there would be a massive groundfight (in Iraq) unless the Yanks decided to go nuclear in the early stages. Don't put that past them: they – and only they – have descended to that level of indiscriminate barbarism before.
But to the extent that members of the Rastus Administration want to be able to travel abroad after Rastus' term finishes, it would be best not to be part of the war-crimes-committing part of a failed Empire. (Ask Dick Cheney, Don Rumsfeld and George Bush about their travel plans: they spend a good deal of their planning time trying to ensure that they will not be arrested for war crimes).
So… in point of fact, everybody (financial markets included) are actually assuming that neither the Yanks, nor the Red Sea Pedestrians are prepared to kick things off.
Guess what? I still think someone somewhere will do something monumentally fucking stupid. That's what happens when people are not subject to bearing the brunt of the consequences of their actions: it doesn't matter if it's the health budget, city planning, infrastructure spending, or international diplomacy – if the individuals involved don't face the consequences of their action, they are prone to overreach, misadventure and downright fucking idiocy. (examples: Iraq, Afghanistan… and the stupid fat mole who was blocking the aisle this morning in CostCo: the architects of each of those disasters get to share the bill with taxpayers).
There are people I know who would not trust the government – ANY government – to run a health care system… but let the politicians decide they are going to have a war, and these same people think you ought to trust the political class absolutely. What. The. Fuck.
Every major war of the 20th century was unnecessary; so far, the two decent-sized shows of the 21st century (plus Libya) were likewise unnecessary. But once the politicians get backed into a corner, they are ALWAYS prepared to flood the drains with blood and set fire to the landscape – to do otherwise might result in not being taken seriously.
Anyway… that's by way of introduction to the fact that I'm now back at the helm. I'm repairing the Rant engines (OzRant seems to be back to tors; USRant is still a bit fucked) and getting ready to unleash madness again. I know I said that some months ago, but circumstances intervened – the exigent circumstance that emerged is now definitively over and done with and can never darken my door again.
I'm also hiving off a decent chunk of the back-end so that it runs remotely: it had got to the stage where it slowed the entire site down for hours at a time (which in turn made some important things go "phut").
Lastly, but not leastly: How fucking good is Housos, uleh? I thought Paulie "Superchoc" Fenech would have a hard time making a show better than 'Fat Pizza', but he has outdone himself – the big dickhead.
OK, so to date this week the observant Reader will notice an absence of the usual congealed lumps of Award-Ready™ brilliance. You will have noticed instead, a persistent 500 – the always-hated server configuration error.
Some of those among you – ye of little faith – may have thought that your Beloved GT had been the victim of counter-#AntiSec DDoS or infiltration. (Two of my chums – who should know better – rang me and asked if the mentatScape had been cyber-raeped… that's the only way I became aware of the issue).
No, no, dearest Reader… it was simpler and more irritating.
You see, unbeknownst to your Beloved GT, the server-wights went and upgraded Apache (getting rid of the hated Lightspeed HTTPD which had prevented the use of keepalive)… which then blew up several config elements – most notably changes to PHP's include_path that had previously been invoked in .htaccess files.
Without the ability to modify the default include_path, a bunch of scripts fail (because they include files that are in a different directory not on the default path). These are scripts that run every day, and build the data and do the calculations and write the generated text for the Rants – and they can't even be run by hand.
There are two possible solutions: one is to go through 80 scripts and insert a one-liner that sets the include path… the other is to find a less intrusive way to unfuck the monkey.
Your Beloved GT is NOT going to be copypasta-ing a variety of one-line ini_set() commands into 80 scripts… I'll tell you that for free. In all likelihood I will get everything fixed by the time of tomorrow's OzRant.
Yes, it's that time of the year again – time to gin around with the layout. The upside of this is that if I get it done right – migrate the presentation layer to a pure-as-snow HTML5/CSS3 thing – it should reduce page load times significantly.
See, CSS3 promises to enable us to do away with some of the kludge that used to be needed to do anything remotely nice. For example, it is possible to get rid of all the nested-div tripe that is involved in having rounded corners and dropshadows; in the past that was possible only by having either nested divs or a whacking great one-size-fits-all background image.
If your browser is standards-compliant, you will see some sexah stuff in the blockquote that follows: if you're using a shit-awful browser (if your browser was made by Microsoft, for example) then it's your fault if you think that all web pages render as unsightly messes.
Suppose I were to invent a new word, "zaxlebax," and define it as "a metallic sphere, like the Washington Monument." That's the definition — "a metallic sphere, like the Washington Monument. " In short, I build my ill-chosen example into the definition. Now some linguistic subgroup might start using the term "zaxlebax" as though it just meant "metallic sphere," or as though it just meant "something of the same kind as the Washington Monument." And that's fine. But my definition incorporates both, and thus conceals the false assumption that the Washington Monument is a metallic sphere; any attempt to use the term "zaxlebax," meaning what I mean by it, involves the user in this false assumption. That's what Rand means by a package-deal term.
Now I think the word "capitalism," if used with the meaning most people give it, is a package-deal term. By "capitalism" most people mean neither the free market simpliciter nor the prevailing neomercantilist system simpliciter. Rather, what most people mean by "capitalism" is this free-market system that currently prevails in the western world. In short, the term "capitalism" as generally used conceals an assumption that the prevailing system is a free market. And since the prevailing system is in fact one of government favoritism toward business, the ordinary use of the term carries with it the assumption that the free market is government favoritism toward business.
There… dropshadow and rounded corners, all done in CSS.
This is definitely not in the 'eye-candy' department, even though it will result in a cleaner look for the site. This is all about reducing page load times, and (in part) doing some stuff at the client end rather than server-side. Once the entire new framework is in place, page load times ought to plummet.
Example: it sounds like nothing, but have a look at the little blue images behind the section headings on the sidebar. It is only 3.3kB and yet in an uncached serve of the page it holds up the page for 2 whole seconds; it could be entirely replaced by a set of CSS3 statements which would not add any appreciable weight to the CSS file. (That said – the delay is caused by some serve misconfig, I think; the server fails to 'keep alive' the connection, which is very annoying. Still, things that can be done without images, should be.
This post will be the top story for a while – normal market reports will appear below it.
First things first: MarketMentat stands in full support of everything Wikileaks does, and stands for. To that end: MM has a Wikileaks mirror where you, the dear Reader, can view the Wikileaks Cable releases as they happen (to date, less than 1000 of the 251,000 cables have actually been made public).
As youse are probly all aware, Julian Assange has been arrested pursuant to the ludicrous trumped-up charges forced down the neck of the Swedish justice system by Marianne Ny – a woman who got tetchy after the international embarrassment of having the initial warrant rescinded within hours of its issue.
Update: JA was denied bail in the 16th-century set-piece that folks for some reason still think is relevant (yes, OK… the State has all the guns and most of the armed goons – I get that). The Australian government, in its usual display of supine gutless craven power-worship to the 'superpower du jour', did the same thing as they always do – kissed ass and refused to aid its citizen.
While JA is a very important fellow in the global information movement, this is not some silly B-Western where you capture (or kill) the guy with the biggest head-dress and the rest of the braves scatter like chaff in the wind. The Yanks (and governments generally) like to think in those terms when considering their 'enemies' – and yet if you suggested that the US or UK would fall apart if Obama or Cameron were assassinated, folks would think you need your head read.
The political parasite class is fighting what I have referred to as a 5th generation enemy (extending William Lind's taxonomy, which tops out at 4th generation).
Lind's taxonomy is something like:
- Ist gen: line and column. Rebs and Union line up across from each other, and shoot;
- 2nd gen: linear fire and movement (WWI-style set pieces; WWII pitched battles);
- 3rd gen: manoeuvre warfare: Waffen-SS (and to a lesser extent, paratroopers and US Marines); bypass or punch through enemy front lines, then attack from the rear
- 4th gen is decentralised insurgent/guerilla warfare; the enemy is decentralised, there are no pitched battles: all the enemy has to do is wait. (In some sense the American Revolutionary War had some aspects of this – but far better examples are US in VietNam, Russia, US in Afghanistan, US in Iraq.. all losses for the 2nd/3rd gen powers against the 4th gen enemy)
A fifth generation engagement involves the same 'ghostly' hard to grasp decentralised structure, where there is no 'field' of battle; there is no territorial (in the genuine sense – terre or terra – dirt) objective… and a goodly chunk of the enemy are within your own home territory.
The enemy – us – does not require armaments, and their objective is to force civilised behaviour on those who live by scoffing our taxes: by 'civilised' I don't mean polite phraseology or bowing to the Queen; I mean stopping fucking killing peasants in far flung parts of the world as part of some ludicrous pissing contest between fucking megalomaniacs.
As youse will by now be aware, we have been going through a process of adding functionality (and prettiness – of which more later) over the last little while. The innards of the site are now working reasonably well, except that the webhost has a nasty habit of dropping the whole of RantSpace off the innerchoobs entirely.
This sitch has gone on for too long, and your Beloved GT – who, as you know, is a delightful and tolerant man in addition to being handsome – has had about enough.
So today I've decreed that RantSpace will warp over to another host this week. I will try to clarify exactly what day that will be: at this stage it seems likey that it will be Wednesday. The actual site innards have already been sticky-taped into a holding cell, ready for migration, but I'm doing another backup today just to ensure minimal disruption.
There is some small risk that the site will be offline as the migration to the new webhost takes place; it might also be the case that your ISPs DNS routing table is 'slow on the uptake'… but you can take it as read that the RantSpace that you know and love will burst forth again like that little alien thingy out of the guy's guts in the original 'Alien'.
Now as to the new prettier face of the site: I got that theme I mentioned, but there is some small twiddling that has to be done in order to set it up all nice-like. There was a ten-minute period on Friday where the new theme was activated as a test run… you might have seen that, but likely not. If there is downtime in the transition period, I will take the opportunity to change themes as well – so if we wink out of existence, you might not recognise the sexy new-look MarketMentat when it re-appears, but it will be us, again.
I mentioned in yesterday's OzRant that I was going to have a go at improving the screen presentation of our globs of congealed Rant-iness. Specifically, it was beginning to irk me that if one had a screen resolution of 1920-by-whatever, reading the pages caused so much eye travel that it was like watching a tennis match.
Anyhow… so off I went and buggerised around trying to fluidify the layout; as usual, IE (specifically ,anything less than IE7) got in the road, because it fails to support min-width and max-width properties. That's because it's a piece of shit.
So – if you've got a non-compliant browser and a decent-sized screen, you're going to have to put up with the eyeball-traverse; for all other folks, it now ought to present a bit more nicer. (It works on my screen in Opera, IE8, Firefox and Chrome).
As I've said numerous times, I have the design flair of a Springbok front-rower – and because I am aware of that, I am also prone to developing growing nagging doubts about the 'look and feel' of anything that I produce; if I adapt a professionally-designed 'theme', over time I get the feeling that my adaptations are a bit, well, shit. And so it is with the current theme ("Magazine" by StudioPress); I've done some very limited tweaking of the theme, but the mental rot has started to set in.
So off I went and had a gander at some other professionally-developed themes – and I think that this one will be given a trial run in the next little while. I'm also going to ditch my self-produced logo and get something decent (since graphic design prices are plummeting).
Dicking around with the presentation layer is a waste of effort if the content is shit – and conversely there are abundant sites that make it clear that if you've got the right content idea, your presentation can be very basic (ZeroHedge, Mish, TickerGuy, 321Gold) or can even be an absolute schemozzle (4chan). I like to think that the RantScape is doing better-than-OK content wise – so it's not just 'lipstick on a pig'.
Also, since there is so little effort required to make things look decent, one ought to at least have a go.
Well, the membership plugin is in place and activated and nothing blew up.
Now it remains for each Page and Category to have the appropriate permissions sticky-taped into place and for the subscription options page to be updated with new, higher prices.
A short note for users of Opera, Chrome and Internet Explorer who don't log in: the caching mechanism will ensure that you get fast delivery of the homepage, however it will almost certainly be a stale page. If you are logged in, the page will not be cached at all – the caching mechanism only presents cached pages to non-subscribers.
The 'stale cache' problem is not straightforward, even though it is specific to one part of the site – the front page. The cache appears to use different logic for the front page than it does for other sections. For example, this page wot you is readin' right now… well, it will appear in the following places:
- the front page;
- the RSS/Atom feed;
- the blog page; and
- the appropriate tag and Category pages.
If you're not logged in, the cached versions of everything except the front page will show this post. Therein lies the mystery to be resolved today.