If you can get an old computer from a second hand store or cheap of craigslist that will be a good start. If it has a decent graphics card on this list then even better:
https://en.bitcoin.it/wiki/Mining_hardware_comparisonI built a small budget GPU mining rig for under $200. The most expensive things are the Power Supply which needs to be at least 600w and even more for a dual setup. I am running a single GPU and my power supply was still over $60 but I got an energy efficient one.
I purchased my GPU off of craigslist for $80 and I can get around 300 MHash/sec. I also had to buy a USB stick for around $20, a hard drive will be more expensive. I used a motherboard and ram out of an old computer and if you purchase these will spend at least $100.
I already had some performance computer fans that cost around $20+ a piece.
All in all here are the parts I used:
Mother board with P4 3.2 Ghz chip: $0 from old comp
1 Gig DDR2 Ram: $0 from old comp
600w Power Supply: $60 from Frys
Computer Power Swich; $0 from old comp
Performance Fans: $20 a piece
16 Gig USB Stick: $20
HD Radeon 6870: $80 off Craigslist
Total: $180~
That's about what you're going to spend on a starter setup and it would be difficult to get it under $100. Even at 300 MHash/sec I'm only making about .01 BTC per day. At this point you might be better off BUYING 1 BTC and waiting till you have enough money for an ASIC.