The Gemara posits the principle that "Yisrael Af Al Pi She'hata Yisrael Hu" – a Jew does not lose his status as a Jew even after he sins. As a general rule, then, a Jew who violates Torah law is nevertheless considered a full-fledged Jew in all respects.
There are, however, a number of exceptions. One is a person who commits sins 'Le'ha'ch'is" – with the specific intention to anger G-d. If a person not only transgresses the Torah, but has decided to spitefully reject G-d, and he thus commits Torah transgressions out of anger and resentment – and not due to convenience, for money, or due a lack of self-discipline – then he forfeits his status as a Jew. Such a person, therefore, cannot be counted toward a Minyan. (A fascinating story is told of Rav Levi Yishak of Berditchev, who once saw a Jewish fellow whom he knew eating pork, and he wished him, "Bon Appetit," that he should enjoy it. He explained to his students that this individual suffered from a certain condition that resulted in pain in his stomach when he ate pork, but he nevertheless ate it out of hostility toward religion. Rav Levi Yishak extended to him the wish that he should eat it for the pleasure that it brings, and not out of resentment, so that he would at least retain his status as a Jew.)
Another exception – which is far more common – is Shabbat desecration. Since Shabbat desecration is equated with idol-worship, a public Shabbat desecrator is, in certain respects, considered not Jewish. The Hafetz Haim explained this Halacha by way of an analogy to a person who walks by a store which is currently closed. If the sign is still up, then the person will likely assume that the store is still in business, and has closed only temporarily. Once, however, the sign has been taken down, the person will conclude that the store has closed permanently. The Torah refers to Shabbat as an "Ot," a "sign." As long as a person observes Shabbat, he demonstrates that he's "open for business," that he's still "in the game," as it were, even if in other ways he might falter. Therefore, a public Shabbat violator – in principle – is not considered a Jew with respect to certain Halachot, such as being counted toward a Minyan.
This applies to public Shabbat violators regardless of the reason why they desecrate Shabbat – whether it's for ideological reasons, out of anger toward G-d, or because of convenience or an unwillingness to make the sacrifices that Shabbat observance requires.
The Poskim dispute the question of how often one must publicly violate Shabbat to be disqualified from counting toward a Minyan. Some maintain that this disqualification applies only to habitual Shabbat violators, but others contend that even if a person publicly violated Shabbat just once, he cannot be counted toward a Minyan. This is the view accepted by Hacham Ovadia Yosef.
It goes without saying that once a person has performed Teshuba and committed to Shabbat observance, he regains his status as a full-fledged Jew and may be counted.
Hacham Ovadia Yosef ruled that one becomes disqualified only by publicly committing an act that is forbidden on Shabbat by force of Torah law. This includes driving, cooking, writing, and carrying in a public domain. If, however, a person publicly commits acts which are forbidden on Shabbat only by force of Rabbinic enactment, he may be counted toward a Minyan.
The act of Shabbat desecration must be committed publicly – meaning, in the presence of ten men, the violator included. The Mishna Berura ruled that the act does not need to have been committed publicly, as even if it becomes known to ten people, this qualifies as a "public" act. Hacham Ovadia, however, disagrees, and rules that a person is not considered a public Shabbat violator unless he committed a forbidden act that was seen by ten men at the same time.
(Incidentally, these conditions apply also with regard to the prohibition against drinking wine handled by a public Shabbat violator (unless the wine has been previously boiled). This applies only to somebody who committed an act of Shabbat desecration in the presence of ten men.)
Hacham Ovadia made a very significant exception to this Halacha, addressing the situation of generally G-d-fearing Jews who work on Shabbat. It was quite common in certain periods that Jews would come to the synagogue, recite Kiddush and eat a Shabbat meal, but then go to work, succumbing to the anxiety of otherwise being unable to support their families. Hacham Ovadia ruled that although such people act incorrectly, nevertheless, since in their eyes, they violate Shabbat under duress, they are not considered intentional Shabbat violators. Hence, they may be counted toward a Minyan (and wine which they handled remains permissible).
Additionally, the Mishna Berura writes that if a person violates Shabbat in public, but is too embarrassed to do so in the presence of a Rabbi, then he is not considered a public Shabbat violator with respect to this Halacha. The fact that he still experiences shame shows that he recognizes the sanctity of Shabbat, and so he does not attain the status of a flagrant desecrator who may not be counted toward a Minyan. However, Hacham Ovadia clarified that this applies only if we are certain that this individual would not desecrate Shabbat in view of his Rabbi.
Much has been written about whether and how these Halachot apply in our generation.
In the case of a person who was raised without a religious education, there is a general consensus that we apply the rule of "Tinok She'nishba" (literally, "an infant who was taken captive"), which excuses from liability a sinner who was never taught about Halachic observance. If someone was not taught to observe Shabbat as Halacha requires, then he is not held accountable for his failure to do so. As such, he does not have the status of a flagrant Shabbat desecrator. Some extend this rule further, asserting that even if a person learned about Shabbat, he cannot be considered a Shabbat desecrator if he was not made aware of the severity of Shabbat. There are people who received some level of Jewish education but their training was not grounded in Halacha, and they were thus never taught about the various Shabbat prohibitions and how they constitute capital offenses. These people, too, might not be considered flagrant Shabbat desecrators.
An even more drastic view was advanced by the Hazon Ish (Rav Avraham Yeshaya Karelitz, 1878-1952). He contended that a person does not attain the status of a public Shabbat desecrator unless he was reprimanded for violating Shabbat and then ignored the rebuke he received. Effective criticism is a difficult art, a skill which the vast majority of people do not have in our time. As such, the Hazon Ish wrote, violators can be assumed to not have ever received proper rebuke for their acts of desecration, and they therefore do not have the status of public Shabbat desecrators. This novel ruling of the Hazon Ish should seemingly allow any Shabbat violator to be counted toward a Minyan nowadays, given the assumption that proper rebuke was never administered.
However, Rav Yisrael Bitan noted that one might question this conclusion, in light of the fact that technology has made knowledge about Torah law readily available to all, and, moreover, there are many Jews who were raised in Torah homes and received a proper religious education, and then decided to abandon halachic observance. It seems difficult to exclude such people from the category of intentional Shabbat desecrators. Rabbi Bitan therefore concludes that every community Rabbi must determine the policy for his congregation, given the different opinions that exist and different realities in each community.
It must be emphasized that we speak here only of being counted toward the minimum quorum of ten men. By no means does Halacha disallow a Shabbat violator from entering a synagogue and participating in the Minyan. Personally, I went into the rabbinate specifically for such Jews, to help Jews grow in their observance, each on their level and in a way that suits them, irrespective of one's current level of commitment. Those who do not properly observe Shabbat should be at least as welcome in the synagogue as fully-observant Jews, if not more so.
I am reminded of a story told by Hacham Baruch Ben-Haim of a person who once came to pray in Congregation Shaare Zion and was given an Aliya, and afterward, somebody else approached Hacham Baruch to object. He said that he knew for a fact that this individual who was given an Aliya regularly shaved with a razor – a strict Torah violation – and thus should not be given the honor of being called to the Torah.
"Let me tell you a story," Hacham Baruch replied. "Many years ago, there was somebody who came to shul who was not at all serious about Halachic observance. We welcomed him very warmly, and gave him an Aliya. There were those who objected and were angry at me. But gradually, this fellow became more involved, attending prayers and classes, and building a relationship with me and other Rabbis. He ended up raising a fully observant home, and all his children are strictly observant.
"As it so happens," the Hacham continued, "that man was your father. You would not be in the synagogue today if I had followed the policy you are advocating."
This should be our attitude toward our fellow Jews who are not as observant as we would want. While there is some question about whether or not they may be counted toward the minimum required amount of ten men, and, as we saw, different opinions exist, there is no question about whether they should be warmly welcomed in our synagogues. Our institutions must be inviting for all Jews, regardless of their level of observance, so everyone can grow in a way and at a pace that is appropriate for them.