Go Back   Rage3D » Rage3D Discussion Area » Community and Site Discussions » Front Page News
Rage3D Subscribe Register FAQ Members List Calendar Mark Forums Read

Front Page News News and Rage3D articles as it appears on the frontpage.

Reply
 
Thread Tools Display Modes
Old Jul 11, 2010, 11:45 AM   #1
Advertisement (Guests Only)

Login or Register to remove this ad
caveman-jim
Deposed King of Rage3D
 
Join Date: Oct 2003
Posts: 48,999
caveman-jim doesn't need no stinkin' badgescaveman-jim doesn't need no stinkin' badgescaveman-jim doesn't need no stinkin' badgescaveman-jim doesn't need no stinkin' badgescaveman-jim doesn't need no stinkin' badgescaveman-jim doesn't need no stinkin' badgescaveman-jim doesn't need no stinkin' badgescaveman-jim doesn't need no stinkin' badgescaveman-jim doesn't need no stinkin' badges


Hard Drive At 21 Petabytes, Facebook Has Largest Known Hadoop Distributed File System Cluster

The Yahoo-started HDFS system is an open source file system designed to use commodity hardware. A part of the Apache Hadoop Core project, it is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high-throughput access to application data and is suitable for applications that have large data sets - like the massive amounts of information generated by Facebook users.

The Datawarehouse Hadoop cluster at Facebook has become the largest known Hadoop storage cluster in the world. Here are some of the details about this single HDFS cluster:
  • 21 PB of storage in a single HDFS cluster
  • 2000 machines
  • 12 TB per machine (a few machines have 24 TB each)
  • 1200 machines with 8 cores each + 800 machines with 16 cores each
  • 32 GB of RAM per machine
  • 15 map-reduce tasks per machine

That's a total of more than 21 PB of configured storage capacity! This is larger than the previously known Yahoo!'s cluster of 14 PB.

Facebook engineers work closely with the Hadoop engineering team at Yahoo! to push Hadoop to greater scalability and performance. Facebook has many Hadoop clusters, the largest among them is the one that is used for Datawarehousing. Here are some statistics that describe a few characteristics of the Facebook's Datawarehousing Hadoop cluster:
  • 12 TB of compressed data added per day
  • 800 TB of compressed data scanned per day
  • 25,000 map-reduce jobs per day
  • 65 millions files in HDFS
  • 30,000 simultaneous clients to the HDFS NameNode


The information is being shared to help answer questions for wild hairy frontiersman (also known as Linux developers) trying to improve Linux swapping with a Hadoop workload, Java developers who may attempt to make Hadoop run faster, and GPU architects who would like to port a Hadoop workload to run on GPUs.


Read more at the Hadoop Blog.

Last edited by Android1 : Jul 11, 2010 at 04:57 PM.
caveman-jim is offline   Reply With Quote
Old Jul 11, 2010, 08:48 PM   #2
Rockshocker
Rage3D Veteran
 
Join Date: May 2004
Location: Canada Hamilton
Posts: 1,251
Rockshocker is still being judged by the masses


Default

So I guess that's why facebook opens up and gets information on the screen so fast all the time. I also guess I'm being asked to trust them that any information connected to me isn't being used in anyway for or against me or anyone else.
Rockshocker is offline   Reply With Quote
Old Jul 12, 2010, 04:11 AM   #3
Hatecrime69
Radeon Evergreen
 
Join Date: Jun 1999
Posts: 1,520
Hatecrime69 is still being judged by the masses


Default

just imagine..all that data and all of it so useless
Hatecrime69 is offline   Reply With Quote
Advertisement (Guests Only)
Login or Register to remove this ad
Old Jul 12, 2010, 04:54 AM   #4
Debello
Not built for speed
 
Join Date: Mar 2004
Location: United States Phoenix, AZ
Posts: 6,368
Debello is still being judged by the masses


Default

Quote:
Originally Posted by Hatecrime69 View Post
just imagine..all that data and all of it so useless
Come on then you wouldn't have gems like

Lizz Conroy My babysitter stole my vibrator... and batteries! oh' alien babies in my butt!
__________________
No so patiently waiting for Everquest: Landmark
Debello is offline   Reply With Quote
Old Jul 12, 2010, 05:20 AM   #5
Hatecrime69
Radeon Evergreen
 
Join Date: Jun 1999
Posts: 1,520
Hatecrime69 is still being judged by the masses


Default

ok, so there are a few gems, i'll give you that, but most of it is useless
Hatecrime69 is offline   Reply With Quote
Old Jul 12, 2010, 09:12 AM   #6
gamekill
keep gaming!
 
Join Date: Feb 2003
Posts: 1,185
gamekill is still being judged by the masses


Default

And someday in the future, probably all this horsepower will be in each one's personal computer. Or cellphone. Or implant or whatever.

Remember the building-sized computers of 40 years ago...

__________________
||my games:_www.mynightjobgame.com__www.chameleongame.com__www.minihuntgame.com
|| Witcher 3 ::: Unreal Engine 4
|| MSI Ge62 2QE ::: i7-4720 ::: 16GB DDR3 ::: GTX965m ::: Evo850 500GB ssd
|| ::: New 3DS cfw Luma :::
||____________________________________________________________________________________________
gamekill is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New facebook app, makes Wii buddy system a possibility klutzon Console Gaming 0 Jun 6, 2008 01:04 PM
Looking for a distributed system tray alert system kepler Off Topic Lounge 5 Jan 31, 2006 09:53 AM
What is the best File System for any OS? Axe_Raider Operating Systems 9 Aug 26, 2003 09:43 AM
system 32 file folder CoolNitro Operating Systems 2 Jun 8, 2003 12:54 AM
Re: System File Protection Pegus Legacy ATI Discussion and Technical Support 0 Aug 25, 2000 07:48 AM


All times are GMT -5. The time now is 05:09 PM.



Powered by vBulletin® Version 3.6.5
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
All trademarks used are properties of their respective owners. Copyright ©1998-2011 Rage3D.com
Links monetized by VigLink