From webster's:
Communism: 1 a : a theory advocating elimination of private property b : a system in which goods are owned in common and are available to all as needed
2 capitalized a : a doctrine based on revolutionary Marxian socialism and Marxism-Leninism that was the official ideology of the U.S.S.R. b : a totalitarian system of government in which a single authoritarian party controls state-owned means of production c : a final stage of society in Marxist theory in which the state has withered away and economic goods are distributed equitably d : communist systems collectively
Socialism: 1 : any of various economic and political theories advocating collective or governmental ownership and administration of the means of production and distribution of goods
2 a : a system of society or group living in which there is no private property b : a system or condition of society in which the means of production are owned and controlled by the state
3 : a stage of society in Marxist theory transitional between capitalism and communism and distinguished by unequal distribution of goods and pay according to work done
Based on that, it's a bit tough to say which the Federation is. If you take communism to mean elimination of all private property, it would seem closer to socialism - there is still some ownership of goods - Picard's flute, personal bottles of wine, Sisko's baseball, etc. But not a whole lot. On the communist side, goods are held in common by the Federation and distriubted as needed. You could probably make a case for either.
Sorry for the wild tangent, though I must say I am interested in what people have had to say regarding this.